Corrupt/truncated CacheDir file causes nil-pointer panic (SIGSEGV)

[Noethix55555]

Summary

When CacheDir is enabled, a corrupted or truncated cache file crashes the process with a nil-pointer dereference (SIGSEGV) instead of falling back to a live fetch.

Cause

In httpBackend.Cache (http_backend.go, ~L172-180):

if file, err := os.Open(filename); err == nil {
    resp := new(Response)
    err := gob.NewDecoder(file).Decode(resp)
    file.Close()
    checkResponseHeadersFunc(request, resp.StatusCode, *resp.Headers) // derefs before checking decode err
    if resp.StatusCode < 500 {
        return resp, err
    }
}

The gob decode error is ignored. When the cache file is partial/corrupt, gob.Decode errors out and leaves resp.Headers == nil. The subsequent *resp.Headers then panics. colly has no recover(), so the process (or an async worker goroutine) crashes.

How it happens in practice

The cache file can be left partial/corrupt by an interrupted write, a disk-full condition, a crash mid-write, or external tampering.

Expected behavior

A corrupt cache file should be treated as a cache miss: discard it and perform a live fetch, rather than crashing the process.

Fix

Check the decode error (and nil Headers) before using resp; on any decode error, remove the corrupt file and fall through to a live fetch. PR to follow.