nix-daemon crashes when an S3 substituter uses an SSO profile

[m4dc4p]

When the configured S3 substituter URL specifies profile=<name> and that profile is an AWS SSO profile (sso_session / sso_start_url / sso_region etc.), the nix-daemon process crashes with Nix daemon disconnected unexpectedly (maybe it crashed?).

The crash is triggered solely by the shape of the profile — no valid SSO session, no cached SSO token, no real AWS account, and no real S3 bucket are required to reproduce. Pointing the substituter at a bogus bucket name is sufficient.

A profile with static credentials (aws_access_key_id / aws_secret_access_key) under the same conditions does not crash.

Origin

git bisect pinpointed the bug at commit ec91479 ("libstore: add AWS SSO support for S3 authentication"; originally tagged in 2.34.0). The prior commit (a6eb2e9) does not show the bug. I'm unclear if the bug reproduces on master ( #ff05a94aa) - something else seemed to be blocking the ability to start the daemon.

Steps To Reproduce

1. Add the following to ~/.aws/config (no ~/.aws/credentials changes
   required):

   [profile brokennix]
   sso_session = brokennix-sso
   sso_account_id = 123456789012
   sso_role_name = FakeRole
   region = us-east-2
   
   [sso-session brokennix-sso]
   sso_start_url = https://fake.awsapps.com/start
   sso_region = us-east-2
   sso_registration_scopes = sso:account:access
   
   [profile workingnix]
   aws_access_key_id = nixadmin
   aws_secret_access_key = nixadmin-secret-123

   Note: the SSO profile points at a nonexistent endpoint
   (https://fake.awsapps.com/start) and uses a fake account id and role.
   No cached SSO token under ~/.aws/sso/cache/ is required.

2. Run nix-store --realise against a bogus store path, with the substituter
   pointed at the SSO profile:

   sub='s3://baz?profile=brokennix'
   nix-store -vvvv \
     --realise "/nix/store/00000000000000000000000000000000-nonexistent" \
     --option substituters "$sub" \
     --option trusted-substituters "$sub"

3. Observed: the daemon (worker) crashes:

   …
   (aws:channel) id=0x…: creating new slot 0x…
   (aws:socket-handler) id=0x…: Socket handler created with max_read_size of 16384
   (aws:socket) id=0x… fd=39: subscribing to readable events
   (aws:channel-bootstrap) id=0x…: Setting up socket handler on channel 0x… with handler 0x… on slot 0x…
   (aws:channel) id=0x…: creating new slot 0x…
   closing daemon connection because of an exception
   error: Nix daemon disconnected unexpectedly (maybe it crashed?)
   

4. Run the same command, swapping in the static-credentials profile:

   sub='s3://baz?profile=workingnix'
   nix-store -vvvv \
     --realise "/nix/store/00000000000000000000000000000000-nonexistent" \
     --option substituters "$sub" \
     --option trusted-substituters "$sub"

   Observed: the daemon survives and the command fails cleanly with the
   expected diagnostic:

   error: path '/nix/store/00000000000000000000000000000000-nonexistent' is required, but there is no substituter that can build it
   

Expected behavior

The daemon should not crash. With an SSO-shaped profile whose preconditions
are not satisfied (no cached token, unreachable SSO endpoint, etc.), the AWS
SDK's SSO credential provider is expected to either:

• fail gracefully and skip itself in the provider chain (matching the behavior
  observed with a malformed/missing profile section), or
• surface a normal error such as "could not resolve credentials" that nix
  reports without terminating the daemon.

Either way, behavior should mirror the static-credentials path: the build
should fail with no substituter that can build it, without taking the daemon
down.

Metadata

nix (Nix) 2.34.7

• Platform: macOS 26.3.1 (Tahoe), arm64 (Apple Silicon)

Additional context

• The trigger appears to be the AWS SDK's SSO credential provider being added
  to the chain (logged as Added AWS SSO Credential Provider to chain for profile '<name>') and proceeding through TLS channel setup. Static-key
  profiles never open a TLS connection during credential resolution and thus
  never hit the broken CF/Security state.
• Static-credential profiles, including profiles with empty/missing credential
  fields, do not exhibit the crash — only profiles whose configuration shape
  causes the SDK to instantiate the SSO provider (and thus call
  SSLCreateContext from a forked-without-exec child) do.
• Only the forked worker crashes, not the listener daemon. The listener
  itself remains alive and accepts new connections, but any subsequent
  connection that triggers the same code path will fork a fresh worker that
  also crashes.
• Running the equivalent operation against a local store (--store local,
  no daemon process) succeeds: TLS negotiation completes, credential
  resolution fails cleanly with an authentication error, and the substituter
  falls through normally.

Checklist

• checked [latest Nix manual] ([source])
• checked [open bug issues and pull requests] for possible duplicates

[m4dc4p]

@Mic92 @edolstra This blocks our org from updating past 2.33.x. Any thoughts? I think its pretty easy to reproduce.

[xokdvium]

This isolates the bug to the fork-without-exec
code path on macOS.

This is only observable in macos? I recall @edolstra mentioning some crash reports under similar conditions.

[m4dc4p]

@xokdvium I haven't tried on Linux, so unclear if this is macos only or not. I removed that bit of the description.

[m4dc4p]

Bump ....

[Mic92]

Can't reproduce on Linux. Do you have a macOS, we could use for testing? Otherwise we need actual crashdumps from macOS and logs from the nix-daemon service. Might be related to the tls stack on macOS that is different from Linux (openssl)

[Mic92]

I was able to reproduce the crash on macOS. looks like the tls stack is not fork-safe like many things in apple frameworks.

[Mic92]

Don't think this is an easy fix, it seems to do some fork-unsafe IPC stuff on each tls connection. This would require the deamon to add re-exec after fork.

[xokdvium]

This would require the deamon to add re-exec after fork.

I was planning on doing that for next release for unrelated reasons (getting more benefit from ALSR).

[Mic92]

#16094

[Mic92]

This would require the deamon to add re-exec after fork.

I was planning on doing that for next release for unrelated reasons (getting more benefit from ALSR).

Sounds fine, but I don't think we want to backport this change to fix older versions of nix. Switching to s2n from appleframework seems to be less invasive for now.

[m4dc4p]

Thanks for the quick action on this. I'm not unable to reproduce locally w/ 2.34.6 🤯 anymore, likely due to a MacOS upgrade from Tahoe 26..3.1 to 26.5 (I found this on a prior version, 26.3.1).

However, I'll run with PR for now and see if i notice any problems!

[Mic92]

@m4dc4p I was able to produce the crashes before my fix.