[Infra cache] Upstream servers do not recover after connectivity loss

[Shmillerov]

Environment

• Version: 1.25.0
• Configuration: infra-host-ttl: 900, infra-cache-max-rtt: 120000, infra-keep-probing: no (defaults)
• OS: Ubuntu 22.04.5 LTS

Background

1. Unbound lost outbound internet access for ~10 minutes due to infrastructure issues
2. During this period upstream servers reached rto = 120000 (infra-cache-max-rtt)
3. Connectivity was restored, SERVFAILs stopped, unbound resumed normal operation
4. After 5 days unbound still have expired rto 120000 state for 11 of 13 root servers, so unbound only queries 2 of them. Root servers are used here as the most visible example, the same situation for any zone

$ unbound-control dump_infra | grep " \. "
2a02:5501:0:6000::c707:5b0d . ttl 577 ping 16 var 53 rtt 228 rto 228 tA 0 tAAAA 0 tother 0 ednsknown 1 edns 0 delay 0 lame dnssec 0 rec 0 A 0 other 0
2a02:5501:0:6000::ca0c:1b21 . expired rto 120000
2a02:5501:0:6000::c661:be35 . expired rto 120000
2a02:5501:0:6000::c03a:801e . expired rto 120000
2a02:5501:0:6000::c070:2404 . expired rto 120000
2a02:5501:0:6000::c024:9411 . expired rto 120000
2a02:5501:0:6000::c005:5f1 . expired rto 120000
2a02:5501:0:6000::c021:40c . expired rto 120000
2a02:5501:0:6000::c100:e81 . ttl 577 ping 23 var 65 rtt 283 rto 1132 tA 2 tAAAA 0 tother 0 ednsknown 1 edns 0 delay 0 lame dnssec 0 rec 0 A 0 other 0
2a02:5501:0:6000::aaf7:aa02 . expired rto 120000
2a02:5501:0:6000::c707:532a . expired rto 120000
2a02:5501:0:6000::c629:4 . expired rto 120000
2a02:5501:0:6000::c0cb:e60a . expired rto 120000

Actual behavior

No automatic recovery. infra_get_lame_rtt() assigns sel_rtt = 119000 ms to expired entries. iter_server_selection only considers servers within RTT_BAND = 400 ms of the best live server, so expired servers never enter the candidate pool -> never probed -> never recovers.

Issues:

• After the internet was restored, unbound randomly selected some server, but only some of them
• Selected servers are not necessarily the fastest
• Other servers do not participate in DNS resolution - there is no balancing between upstreams
• Monitoring is difficult - we don't have statistics on other upstreams

Setting infra-keep-probing: yes was tested both at runtime via unbound-control set_option and locally from the start in config.
With the option enabled, SERVFAILs stopped during the outage once all servers hit rto = 120000; unbound kept sending single probe queries instead of failing immediately.
However, after connectivity was restored, servers with expired rto 120000 still did not recover over time.

Proposed fix

1. Active background probing of expired entries independent of query selection. Fix what infra-keep-probing was intended to do but currently doesn't.

2. Modify infra_get_lame_rtt() to return UNKNOWN_SERVER_NICENESS instead of still_useful_timeout() for expired records with max rto. This makes them fit within RTT_BAND and periodically receive queries, naturally triggering recovery.

Workaround

Flush infra cache:

unbound-control flush_infra all

But this is manual action, not a solution. I think that self-healing is expected behavior and should work without any manual actions

[Shmillerov]

Hello. I'll raise an issue.
Is the current behavior expected?
Do you think this is an issue or I'm missing some configuration?

[wcawijngaards]

Unbound has to back off, from servers that do not respond well. The situation where there is connectivity loss looks similar. The option can be fixed as proposed to probe more often, that is not by design that it does not. So to return the unknown server niceness value, but only when the option is in use. Without the option, the behavior is intentional, to stop sending traffic to servers that are down.

[Shmillerov]

Unbound has to back off, from servers that do not respond well. The situation where there is connectivity loss looks similar. The option can be fixed as proposed to probe more often, that is not by design that it does not. So to return the unknown server niceness value, but only when the option is in use. Without the option, the behavior is intentional, to stop sending traffic to servers that are down.

To confirm I've got this right: the fix would be to make infra_get_lame_rtt() return UNKNOWN_SERVER_NICENESS instead of still_useful_timeout() for expired records at max rto, but only when infra-keep-probing is set to yes.

Is that right?

[wcawijngaards]

It sounds allright, and for only when the option is set, and then it would probe more often, seems like it can be an idea.

[myarcana]

I've been getting frustrated all the time recently by problems to do with this issue (I guess), on a spotty internet connection.

Apparently during the kinds of packet loss my internet connection was experiencing intermittently, unbound was marking authoritative name servers as unreachable. That they were "unreachable" was getting stored in the infra cache for 15 minutes! So when my internet connection finally became much better again, I still couldn't use the internet, because unbound was just serving cached SERVFAILS for everything.