feat(mrf): V4 response migration — PR 2/5 BE V4-native + V3 wire shim

[scottheng96]

Problem

PR 1 added the V4 building blocks (types, utils, validators) as pure additions. This PR is the internal pipeline flip — the BE stops treating V3 as the working format and instead handles every MRF submission as V4 end-to-end, with the wire-side V3 shape adapted into V4 at the receiver.

Two compatibility constraints to preserve:

• Stale FE clients (cached SPA bundles, open browser tabs) will keep sending V3 shape after BE deploy. They must continue to work seamlessly.
• Webhook consumers parse the encrypted payload as V3 (mrfVersion: 1). They cannot break.

Solution

Four commits on top of feat/mrf-v4-pr1-foundation:

1. feat(mrf): migrate submission pipeline to V4 response types end-to-end — cherry-pick of the original V4 wiring commit from feat/mrf-v4-responses. Receiver types req.body.responses as FieldResponsesV4; scanAndRetrieveAttachments, validateMultirespondentSubmission, encryptSubmission, handleNdiResponses, email utils, and multirespondent-submission.service.ts all consume V4 throughout. mrfVersion defaults to 2. Legacy in-flight chains with mrfVersion: 1 in DB still complete via the existing adaptV3ToV4 call in validateMultirespondentSubmission (unchanged).

2. fix(mrf): restore buildMrfResponseJson and getFormDelimiter for V4 — these helpers were added to develop after the reference branch forked. The cherry-pick dropped them; the service still calls them. Restored as V4-native (the rewrite delegates to getQuestionAnswerPairsForMultipleFields). Note: the delimiter param is accepted for caller compatibility but is currently a no-op — table-cell separator is hardcoded to '; ' in the V4 utils. Restoring admin metadata.delimiter customisation is a follow-up.

3. feat(mrf): adapt stale V3 MRF submissions to V4 at the receiver — V3→V4 shim at the receiver boundary. Triggered by body.version >= 3 && < 4, runs the SDK's adaptV3ToV4 in place, then bumps body.version to 4 so downstream code can uniformly treat the body as V4. Runs before addAttachmentToResponses so attachment buffers land in V4-shaped answers. Emits a WARN log on each adapted submission for stale-FE monitoring during rollout. Adapted responses have question: '' (form definition not loaded at receiver layer); the existing downstream fallback response.question || formField.title handles this cleanly.

4. feat(mrf): webhook V3 encryption path + restore [Verified] Q/A prefix — two related compatibility tweaks:

   • Forms with formDef.webhook?.url set encrypt as V3 (mrfVersion: 1) via adaptV4ToV3 right before cryptoV3.encrypt. In-process state (encryptedPayload.responses, email Q/A, NDI merging) stays V4 throughout — only the encrypted blob differs. Webhook consumers continue to parse V3 unchanged.
   • Restore the [Verified] prefix on email/mobile question titles in Q/A pairs used for email body / PDF rendering — this UX hint was dropped by the wiring commit. Toggle via the new includeVerifiedPrefix option (default true). buildMrfResponseJson passes false so the JSON dump carries raw question text.

Status: Being soaked on a staging environment in parallel with this review. Marked as draft until staging confirms parity with V3 production behavior.

Migration roadmap (5 PRs):

1. PR 1 — Foundation: V4 parsed types, V4 utility functions, V4 validator variants added alongside existing V3. Pure additions.
2. PR 2 (this) — BE V4-native end-to-end: Cherry-pick the V4 wiring commit (receiver/scan/validate/encrypt/email/NDI all use V4). Add a V3→V4 receiver-side shim so stale FE bundles (still sending V3) continue to work during the rollout window. Webhook forms keep V3 encryption (mrfVersion: 1) via V4→V3 adapter at encryption time only — preserves the V3 webhook payload contract.
3. PR 3 — FE sends V4: Replace createResponsesV3 with a V4-thin wire builder (FE sends fieldType + answer only; BE enriches metadata). FE bumps submission version to 4.
4. PR 4 — Cleanup: Delete V3 validators, V3 parsed types, V3 logic eval (BE), and the FE V3 builder. V4 is the only path.
5. PR 5 — Remove V3 receiver shim: Drop the V3→V4 shim once telemetry confirms V3 wire traffic has fully drained (stale browser tabs / cached SPA bundles).

Risk worth flagging. The original 5-PR plan had an intermediate step where the BE accepted V4 from wire but routed through the V3 pipeline (adapt V4→V3 at entry). That gave the V4 utilities first exercise on a non-critical path before the internal flip. By skipping it, the V4 validators / Q/A helpers / encryption branch get their first prod exercise on every MRF submission the moment this PR lands. Staging soak is the primary mitigation; rolling back is just reverting the merge.

Breaking Changes

• No external contract changes. Wire format: V3 submissions still accepted via the receiver shim. Webhook payload: still V3 (mrfVersion: 1) when formDef.webhook?.url is set. mrfVersion: 2 is new on the DB record for non-webhook forms.

Tests

TC1: V4-shaped FE submission (the happy path PR 3 will enable)

• Submit a multi-step MRF with body.version: 4 and V4-shaped responses
• Verify each step succeeds and responses persist correctly
• Verify final stored mrfVersion is 2
• Verify completion email contains correct Q/A pairs

TC2: Stale-FE V3 submission (receiver shim)

• Submit an MRF with body.version: 3 and V3-shaped responses (string answer, etc.)
• Verify submission succeeds end-to-end
• Verify a WARN log is emitted: "Adapting V3 MRF submission to V4 — client is on a stale build and should refresh"
• Verify the final stored mrfVersion is 2 (non-webhook) or 1 (webhook)
• Verify attachment fields persist content + filename correctly

TC3: Webhook form encryption fallback

• Configure an MRF form with a webhook URL
• Submit the form (any version)
• Verify the stored mrfVersion is 1
• Decrypt the stored encryptedContent and verify the shape is V3 (answer as string for text, addressSubFields for address, etc.)
• Verify the webhook fires and consumer receives a V3-shape payload

TC4: [Verified] prefix on signed email/mobile

• Submit an MRF with a verified email field (carries signature on the answer)
• Verify the completion email body shows [Verified] <field title> for that field
• Verify the email's attached response JSON (responseJson) does not include the [Verified] prefix — raw title only

TC5: Multi-step continuity with attachments

• Step 1: submit with an attachment field
• Step 2 (different respondent): open the form, confirm step-1 attachment metadata is pre-filled
• Submit step 2
• Verify the attachment is downloadable from the final submission

TC6: Legacy in-flight chain (mrfVersion: 1 in DB)

• Find or seed an MRF submission record with mrfVersion: 1 (pre-migration)
• Submit the next workflow step
• Verify validateMultirespondentSubmission calls adaptV3ToV4 on the previous-step responses before comparison (no field comparison failures)
• Verify the step completes successfully

TC7: Build & static checks

• CI typecheck passes (pnpm --filter formsg-backend build — verified locally: clean)
• Lint passes

[Copilot]

Pull request overview

This PR flips the backend Multi-Respondent Form (MRF) submission pipeline to be V4-native internally (types/validation/Q&A helpers/encryption flow), while preserving compatibility for (a) stale FE clients still sending V3 and (b) existing webhook consumers that expect V3-encrypted payloads.

Changes:

• Migrates MRF request/processing types and field validators to operate on V4 response shapes end-to-end, including attachment scanning and Q/A extraction utilities.
• Adds a receiver-side V3 → V4 shim for version >= 3 && < 4 submissions so stale FE clients continue working.
• Preserves webhook compatibility by encrypting webhook forms as V3 (via V4 → V3 adaptation at encryption time) while keeping in-process handling in V4.

Reviewed changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/shared/package.json	Adds CJS require export for ./constants/form to keep require('formsg-shared/constants/form') working.
packages/sdk/cjs-entry.cjs	Exports callable default with named exports attached for CJS consumers.
apps/backend/src/types/api/submission.ts	Introduces V4 parsed/clear response types, including receiver-enriched V4 attachment answers.
apps/backend/src/types/api/multirespondent_submission.ts	Switches MRF API body and DTO response types from V3 to V4.
apps/backend/src/app/utils/response-v4.ts	Adds V4 response equality helper (used in non-editable field checks).
apps/backend/src/app/utils/field-validation/validators/yesNoValidator.ts	Adds Yes/No validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/uenValidator.ts	Adds UEN validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/textValidator.ts	Adds short/long text validators for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/tableValidator.ts	Adds table validator for V4 keyed-row table answer shape.
apps/backend/src/app/utils/field-validation/validators/signatureValidator.ts	Adds signature validator for V4 signature answer shape.
apps/backend/src/app/utils/field-validation/validators/sectionValidator.ts	Adds section validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/ratingValidator.ts	Adds rating validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/radioButtonValidator.ts	Adds radio validator for V4 { value, isOthersInput } answer shape.
apps/backend/src/app/utils/field-validation/validators/numberValidator.ts	Adds number validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/nricValidator.ts	Adds NRIC validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/mobileNoValidator.ts	Adds mobile validator for V4 verifiable answer shape and V4 signature validation hook.
apps/backend/src/app/utils/field-validation/validators/homeNoValidator.ts	Adds home number validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/emailValidator.ts	Adds email validator for V4 verifiable answer shape and V4 signature validation hook.
apps/backend/src/app/utils/field-validation/validators/dropdownValidator.ts	Adds dropdown validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/decimalValidator.ts	Adds decimal validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/dateValidator.ts	Adds date validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/countryRegionValidator.ts	Adds country/region validator for V4 response shape.
apps/backend/src/app/utils/field-validation/validators/common.ts	Adds V4 signature validator constructor for verifiable fields.
apps/backend/src/app/utils/field-validation/validators/childrenValidator.ts	Adds V4 “always pass” children validator (MRF not using children fields).
apps/backend/src/app/utils/field-validation/validators/checkboxValidator.ts	Adds checkbox validator for V4 checkbox answer shape.
apps/backend/src/app/utils/field-validation/validators/attachmentValidator.ts	Adds V4 attachment validator for nested answer.content/filename.
apps/backend/src/app/utils/field-validation/validators/addressValidator.ts	Adds address validators for V4 address subfield answer shape.
apps/backend/src/app/utils/field-validation/index.ts	Adds validateFieldV4 path and V4 validation error mapping.
apps/backend/src/app/utils/field-validation/answerValidator.factory.ts	Adds V4 field-response validator factory wiring.
apps/backend/src/app/modules/submission/submission.utils.ts	Adds V4 attachment type guard used by the MRF pipeline.
apps/backend/src/app/modules/submission/submission.service.ts	Adds GuardDuty scan/download helper for V4 attachment response shape.
apps/backend/src/app/modules/submission/submission.errors.ts	Adds ValidateFieldErrorV4 error class.
apps/backend/src/app/modules/submission/receiver/receiver.utils.ts	Adds receiver-side V3→V4 MRF shim and updates attachment injection for V4 responses.
apps/backend/src/app/modules/submission/receiver/receiver.types.ts	Updates receiver body type-guard for v3+ submissions to V4 response record.
apps/backend/src/app/modules/submission/receiver/receiver.service.ts	Plumbs receiver-side shim into multipart parsing flow and updates typings.
apps/backend/src/app/modules/submission/receiver/receiver.middleware.ts	Updates multirespondent receiver middleware request body type to V4.
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.utils.ts	Migrates MRF utilities to V4 response shapes (validation, Q/A extraction, response JSON).
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.types.ts	Updates MRF types to V4 response types and V4 stripped attachment type.
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts	Migrates service-layer typing to V4 responses (e.g. approval checks and email flows).
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.middleware.ts	Main MRF pipeline flip to V4, including webhook V3 encryption branch and V4 equality comparison.
apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.utils.spec.ts	Updates unit tests for V4 response shapes and new Q/A formatting behavior.
apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.service.spec.ts	Updates service tests to V4 { value } answers for yes/no fields in fixtures.
apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.middleware.spec.ts	Updates middleware tests for webhook V3 encryption branch and V3→V4 previous-response adaptation path.
apps/backend/src/app/modules/spcp/spcp.util.ts	Adds V4 NDI response builder for inclusion in V4 email/encrypted payload handling.
apps/backend/src/app/modules/core/core.errors.ts	Adds new error code for V4 field validation failures.