AddHeadersToDownstream will produce NullReferenceException

[bobbydog]

I use a PreAuthorizationMiddleware to fact-check the token is validated, when ran several hours, it produce NullReferenceException and request return 500 errcode.

public class TokenValidateMiddleware
{
    public async Task InvokeAsync(HttpContext context,Func<Task> next)
    {
        var rst = await AuthorizationHandler(context);
        if (!rst)
        {
            return;
        }
        await next.Invoke();
    }
    private async Task<bool> AuthorizationHandler(HttpContext context)
    {
        bool isSuccess = false;
        var downRoute = context.Items.DownstreamRoute();
        string refresh = "false";
        //route need to check token
        if (downRoute.AuthenticationOptions != null && downRoute.AuthenticationOptions.AuthenticationProviderKeys != null && downRoute.AuthenticationOptions.AuthenticationProviderKeys.Length > 0)
        {
            var token = context.Request.Headers["Authorization"].FirstOrDefault()?.Split(" ").Last();
            var fact = context.RequestServices.GetService<IHttpClientFactory>();
            if (fact != null)
            {
                using (var httpClient = fact.CreateClient("AuthApi"))
                {
                    try
                    {
                        if (!string.IsNullOrEmpty(token))
                        {
                              //send a http request to know token is removed or ready to timeout
                                if ( removed)
                                {
                                    var error = new UnauthenticatedError("Token is invalidated");
                                    context.Items.SetError(error);
                                    return isSuccess;
                                }
                                else  if (rready to timeout)
                                {
                                    refresh = "true"; 
                                }
                            }
                        }
                        else
                        {
                            var error = new UnauthenticatedError("Token is absent");
                            context.Items.SetError(error);
                            return isSuccess;
                        }
                    }
                    catch (Exception ex)
                    {
                        var error = new UnauthenticatedError("Token is invalidated");
                        context.Items.SetError(error);
                        return isSuccess;
                    }
                }
            }
        }
        var _addHeaders = new List<AddHeader>() { new AddHeader("RefreshToken", refresh) };
        downRoute.AddHeadersToDownstream.AddRange(_addHeaders);
        isSuccess = true;
        return isSuccess;
    }
}

When it run , I can receive response header or the 401 response

Several hours it only return 500 error for all request.

Category: Ocelot.Errors.Middleware.ExceptionHandlerMiddleware
EventId: 0
SpanId: 57ec213c10211302
TraceId: 9e2af5d4000ea8459a97332fb200af47
ParentId: 0000000000000000
ConnectionId: 0HNK0JSC1H4SU
RequestId: 0HNK0JSC1H4SU:00000001
RequestPath: /authapi/Auth/KeepLogin

RequestId: 0HNK0JSC1H4SU:00000001, PreviousRequestId: -
NullReferenceException caught in global error handler!
DONE reporting of a total 0 inner exceptions for request 0HNK0JSC1H4SU:00000001 of the original NullReferenceException below ->

Exception: 
System.NullReferenceException: Object reference not set to an instance of an object.
   at Ocelot.Headers.AddHeadersToResponse.Add(List`1 addHeaders, DownstreamResponse response)
   at Ocelot.Headers.Middleware.HttpHeadersTransformationMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.Security.Middleware.SecurityMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.Multiplexer.MultiplexingMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.Responder.Middleware.ResponderMiddleware.Invoke(HttpContext context)
   at Ocelot.Errors.Middleware.ExceptionHandlerMiddleware.Invoke(HttpContext context)

Files

• Program_main.txt
• ocelot.json

[raman-m]

@bobbydog
How do you use Ocelot? Do you just add a reference to the Ocelot package in an ASP.NET application?

I don't see a forked Ocelot repository in your public repos. How do you develop your project without having the actual code to debug?

[raman-m]

I use a PreAuthorizationMiddleware to fact-check the token is validated, when ran several hours, it produce NullReferenceException and request return 500 errcode.

The code

public class TokenValidateMiddleware
{
    public async Task InvokeAsync(HttpContext context,Func<Task> next)
    {
        var rst = await AuthorizationHandler(context);
        if (!rst)
        {
            return;
        }
        await next.Invoke();
    }
    private async Task<bool> AuthorizationHandler(HttpContext context)
    {
        bool isSuccess = false;
        var downRoute = context.Items.DownstreamRoute();
        string refresh = "false";
        //route need to check token
        if (downRoute.AuthenticationOptions != null && downRoute.AuthenticationOptions.AuthenticationProviderKeys != null && downRoute.AuthenticationOptions.AuthenticationProviderKeys.Length > 0)
        {
            var token = context.Request.Headers["Authorization"].FirstOrDefault()?.Split(" ").Last();
            var fact = context.RequestServices.GetService<IHttpClientFactory>();
            if (fact != null)
            {
                using (var httpClient = fact.CreateClient("AuthApi"))
                {
                    try
                    {
                        if (!string.IsNullOrEmpty(token))
                        {
                              //send a http request to know token is removed or ready to timeout
                                if ( removed)
                                {
                                    var error = new UnauthenticatedError("Token is invalidated");
                                    context.Items.SetError(error);
                                    return isSuccess;
                                }
                                else  if (rready to timeout)
                                {
                                    refresh = "true"; 
                                }
                            }
                        }
                        else
                        {
                            var error = new UnauthenticatedError("Token is absent");
                            context.Items.SetError(error);
                            return isSuccess;
                        }
                    }
                    catch (Exception ex)
                    {
                        var error = new UnauthenticatedError("Token is invalidated");
                        context.Items.SetError(error);
                        return isSuccess;
                    }
                }
            }
        }
        var _addHeaders = new List<AddHeader>() { new AddHeader("RefreshToken", refresh) };
        downRoute.AddHeadersToDownstream.AddRange(_addHeaders);
        isSuccess = true;
        return isSuccess;
    }
}

OK... But why do you think that we should care about your customizations?
Do you really believe that this uncovers a potential bug in the Ocelot library?
Could you please upload the complete solution to your GitHub repository?
That will help me clone and debug it and finally confirm whether a bug actually exists.

---

AuthorizationHandler

Your approach to implementing authorization by placing authentication and authorization logic inside an Ocelot's delegating handler is incorrect. Even token validation should be moved out of the Ocelot application. The purpose of the handler is being misused because you're mixing application responsibilities—this is a concern, since Ocelot's primary responsibility is routing traffic and forwarding API requests and responses.

You should have two routes:

• Route 1 (non-authenticated) – for anonymous traffic. It allows clients to obtain authorization tokens from your custom authentication server. The generated token is then delivered to the client for use in subsequent requests.
• Route 2 (authenticated) – for private or secured traffic. After token validation, the client initiates a second request with the attached token to a private API. That API requires the token for processing and responds with a successful status code if the token is valid.

Having these two routes (one unauthenticated and one authenticated) will allow you to remove your custom AuthorizationHandler, and/or refactor the handler to be simpler and more stable. As a result, your Ocelot application becomes more stable and aligned with its intended purpose.

The ideal solution is having all routes non-authenticated, forwarding tokens as-is. However, in theory, Ocelot can provide some authentication functionality. But this authentication logic should be as simple as possible. It's better not to have any authentication on Ocelot's side at all.

[bobbydog]

The project don't let same account login in twice, when generate the new token, the older one should be invalidated. the PreAuthorizationMiddleware just send the token to the authentication server to get the check result. If I'm not doing this in ocelot, I would place the same logic in every bussiness api as they check the token individually. And when the token expire time come, we should add the header to notice website to refresh token, isn't it easy to do this in ocelot?

[raman-m]

Oops, you don't use delegating handlers just a custom middleware to inject.

And when the token expire time come, we should add the header to notice website to refresh token, isn't it easy to do this in ocelot?

Yes, it is easy. It would also be much easier for us as a development team if you took more responsibility for your customization. We encourage you to read the official documentation rather than assuming there is a bug in your custom solution.

Question: Why should the Ocelot team be responsible for issues in a custom solution that appears to contain bugs?

Your arguments are not convincing:

• There is no complete solution provided. Attached files don't show complete picture.
• There are no clear steps to reproduce the issue.
• There is no forked repository with a prepared branch demonstrating the problem.

At this point, I do not see a bug in Ocelot. What I see is incorrect usage of the product and a disregard for the documentation and best practices.
My main concern is your manipulation of the core DownstreamRoute business object, which is part of the system internals.
You are taking up our time, but I will provide a more detailed explanation later today.

[raman-m]

                              //send a http request to know token is removed or ready to timeout
                                if ( removed)
                                {
                                    var error = new UnauthenticatedError("Token is invalidated");
                                    context.Items.SetError(error);
                                    return isSuccess;
                                }
                                else  if (rready to timeout)
                                {
                                    refresh = "true"; 
                                }

This C# code does not compile. The removed variable is not defined.
if (rready to timeout), this is not valid code; it appears to be placeholder or pseudo-code.

As I understand it, your custom middleware simply refreshes tokens issued by an identity server. Please note that Ocelot is not, and should not be, an identity server responsible for token generation.
This is an architecturally poor solution. It mixes concerns and violates principles such as SOLID and proper microservices design.

The correct workflow that should be implemented on the client side is:

1. The downstream service processes an unauthenticated request by redirecting to an identity server (e.g., returning HTTP 301 or 302).
2. The client receives the redirect response and sends a separate request to the identity server using the URL from the Location header. See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Location
3. The identity server generates an authentication token and returns it to the client. The client then parses, validates, and includes the token in subsequent requests.
4. The client sends requests to Ocelot routes, allowing authentication data (headers, body, query parameters) to be forwarded to downstream services.
5. The downstream services validate the token. Unauthenticated requests are redirected to the identity server, which issues valid tokens.

This is a well-established and widely used approach.

In theory, Ocelot may redirect unauthenticated traffic to an identity server. However, implementing token refresh logic inside Ocelot is a flawed design, given that a dedicated identity server already exists.

I do not understand the rationale behind embedding identity server functionality into Ocelot via custom middleware.
Please review HTTP fundamentals, particularly the Location header, and refer to Ocelot's official authentication documentation: Authentication feature.
Ocelot's role is to authenticate and route requests, but not to generate tokens. Token generation should be delegated to a dedicated identity server.
In limited scenarios (e.g., fully private internal networks), centralizing this logic might be considered, but it doesn't make much benefits. However, exposing such behavior in a public-facing gateway introduces serious security risks. Improper design could potentially expose downstream services to unauthorized access. A-ha-ha! 🙃

😄 Enjoy!

[raman-m]

Why 2 lines? 🤨

        var _addHeaders = new List<AddHeader>() { new AddHeader("RefreshToken", refresh) };
        downRoute.AddHeadersToDownstream.AddRange(_addHeaders);

It could be a single line:

        downRoute.AddHeadersToDownstream.Add(new("RefreshToken", refresh));

[bobbydog]

Many thanks for your suggestion , as I don't quite familar with the PreAuthorizationMiddleware used . At first from the exception occured, I thought there was something wrong at downRoute.AddHeadersToDownstream , not concerned on my custom code, so I didn't show detail for my custom code as some words not make sense.
and here is the whole ocelot-gateway solution
OcelotGateway.zip

[raman-m]

Returning to a "bug" aka exception:

Exception: 
System.NullReferenceException: Object reference not set to an instance of an object.
   at Ocelot.Headers.AddHeadersToResponse.Add(List`1 addHeaders, DownstreamResponse response)
   at Ocelot.Headers.Middleware.HttpHeadersTransformationMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.Security.Middleware.SecurityMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.Multiplexer.MultiplexingMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.DownstreamRouteFinder.Middleware.DownstreamRouteFinderMiddleware.Invoke(HttpContext httpContext)
   at Ocelot.Responder.Middleware.ResponderMiddleware.Invoke(HttpContext context)
   at Ocelot.Errors.Middleware.ExceptionHandlerMiddleware.Invoke(HttpContext context)

The problem is in the HttpHeadersTransformationMiddleware, so look at the current implementation:

Ocelot/src/Ocelot/Headers/Middleware/HttpHeadersTransformationMiddleware.cs

Lines 31 to 58 in 086c7b1

	public async Task Invoke(HttpContext httpContext)
	{
	var downstreamRoute = httpContext.Items.DownstreamRoute();
	var preFAndRs = downstreamRoute.UpstreamHeadersFindAndReplace;
	//todo - this should be on httprequestmessage not httpcontext?
	_preReplacer.Replace(httpContext, preFAndRs);
	_addHeadersToRequest.SetHeadersOnDownstreamRequest(downstreamRoute.AddHeadersToUpstream, httpContext);
	await _next.Invoke(httpContext);
	// todo check errors is ok
	//todo put this check on the base class?
	if (httpContext.Items.Errors().Count > 0)
	{
	return;
	}
	var postFAndRs = downstreamRoute.DownstreamHeadersFindAndReplace;
	_postReplacer.Replace(httpContext, postFAndRs);
	var downstreamResponse = httpContext.Items.DownstreamResponse();
	_addHeadersToResponse.Add(downstreamRoute.AddHeadersToDownstream, downstreamResponse);
	}

You don't use a Debug version of DLLs just a package from NuGet which has Release DLLs only (without debug synbols). But as far as I understood the problematic lines are:

Ocelot/src/Ocelot/Headers/Middleware/HttpHeadersTransformationMiddleware.cs

Line 57 in 086c7b1

_addHeadersToResponse.Add(downstreamRoute.AddHeadersToDownstream, downstreamResponse);

This line calls the AddHeadersToResponse DI-service which responsible for placeholders replacements for headers coming from downstream response to be returned to the client, thus, they are added to HttpResponseMessage object:

Ocelot/src/Ocelot/Headers/AddHeadersToResponse.cs

Lines 19 to 41 in 086c7b1

	public void Add(List<AddHeader> addHeaders, DownstreamResponse response)
	{
	foreach (var add in addHeaders)
	{
	if (add.Value.StartsWith('{') && add.Value.EndsWith('}'))
	{
	var value = _placeholders.Get(add.Value);
	if (value.IsError)
	{
	_logger.LogWarning(() => $"Unable to add header to response {add.Key}: {add.Value}");
	continue;
	}
	response.Headers.Add(new Header(add.Key, new List<string> { value.Data }));
	}
	else
	{
	response.Headers.Add(new Header(add.Key, new List<string> { add.Value }));
	}
	}
	}
	}

This is the Add to Response feature in terms of Ocelot.

The root cause is located on this line:

Ocelot/src/Ocelot/Headers/AddHeadersToResponse.cs

Line 21 in 086c7b1

foreach (var add in addHeaders)

I think the List<AddHeader> addHeaders argument is null and foreach enumerator generates the exception due to absent instance.

I have totally figured out the route cause. You have misunderstood the purpose of both JSON Schema sections of the Headers Transformation feature.
Since you wanna add a header to HTTP request to be forwarded to downstream, right:

        var _addHeaders = new List<AddHeader>() { new AddHeader("RefreshToken", refresh) };
        downRoute.AddHeadersToDownstream.AddRange(_addHeaders);

This is very risky way to manipulate internal collections of DownstreamRoute because it is system internal logic.
Due to changed DownstreamRoute.AddHeadersToDownstream collection the processing DI-service known as IAddHeadersToResponse can't find appropriate header in response headers or AddHeadersToDownstream collection is null.
Finally, headers transformation middleware got broken, and it can't process transformations due to manually added or removed items.
Your approach to manipulate transformation feature is totally wrong!

Possible fixing solutions

1. Stop manipulate DownstreamRoute internal collections! including AddHeadersToDownstream collection. This collection is built by internal IHeaderFindAndReplaceCreator DI-service:
   

   Ocelot/src/Ocelot/Configuration/Creator/StaticRoutesCreator.cs

   Line 103 in 086c7b1

   var hAndRs = _headerFAndRCreator.Create(fileRoute, globalConfiguration);

   
   and initialized in DownstreamRoute here:
   

   Ocelot/src/Ocelot/Configuration/Creator/StaticRoutesCreator.cs

   Line 121 in 086c7b1

   .WithAddHeadersToDownstream(hAndRs.AddHeadersToDownstream)

2. Use JSON schema for headers transformations, especially you must use Add to Request feature!

"UpstreamHeaderTransform": {
  "RefreshToken": "false" // to be replaced
}

And only after that you can think how and where to update the value. The pre-authentication middleware is not right place.

3rd, The best solution

Because you want to add a simple header to HTTP request to be sent to downstream, it is better to add the header on the fly in a delegating handler. This will be the best solution.

public class AuthorizationHandler : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken token)
    {
        // Do middleware work... init refresh variable
        request.Headers.Add("RefreshToken", refresh); // the line does the job we wanted
        var response = await base.SendAsync(request, token);
        // Do post-processing of the response...
        return response;
    }
}

Then update appropriate routes JSON to utilize the delegating handler.

    {
      "DelegatingHandlers": [ "AuthorizationHandler" ], // !!! you must add this
      "DownstreamPathTemplate": "/api/{everything}",
      "DownstreamScheme": "http",
      "DownstreamHostAndPorts": [
        { "Host": "localhost", "Port": 26423 }
      ],
      "UpstreamPathTemplate": "/srmapi/{everything}",
      "UpstreamHttpMethod": ["Get", "Post"],
      "AuthenticationOptions": {
        "AuthenticationProviderKeys": [ "Bearer" ],
        "AllowedScopes": []
      }
    },

I don't recommend to add the delegating handler globally in services. Better to utilize the handler in required routes only.
Hope it should help❗

[raman-m]

@bobbydog wrote on March 18

Many thanks for your suggestion , as I don't quite familiar with the PreAuthorizationMiddleware used.

I see. Your mistake was mixing the functionality of multiple middlewares into a single middleware. Header transformations logic is quite complex, so design mistakes are easy to make.

At first from the exception occurred, I thought there was something wrong at downRoute.AddHeadersToDownstream, not concerned on my custom code, so I didn't show detail for my custom code as some words not make sense.

As I mentioned earlier, using and manipulating the DownstreamRoute object is very risky. You need an excellent understanding of the Ocelot codebase before attempting to override its services.

and here is the whole ocelot-gateway solution: OcelotGateway.zip

Sorry, but I am not going to download and unpack your project in this archive because I review and provide advice only on code available in public repositories which can be cloned. I have already explained and shown the recommended and best solution ☝️

[raman-m]

@bobbydog
And.. What version of Ocelot package do you reference in app?

[bobbydog]

PackageReference Include="Ocelot" Version="24.1.0"

[@bobbydog] What version of Ocelot package do you reference in app?