[Question] OTel span spam

[tvardero]

I have found that FusionCache contributes ~50% of all telemetry spans in the project. Overall spans with op get or set cache are more common than even db calls.

We don't use cache that much, but it already has large impact on the application. We have one cache call for authorization check (on every endpoint), but other than that, we rarely cache anything else in application.

Investigating further I see that one GetOrSet operation generates 7 different span (1 parent and 6 child), depending on how many tags were specified for the cache entry. Having more tags generates more spans.

All child spans differentiate only in fusioncache.operation.key attribute:

We are using self-hosted Sentry and large amount of spans makes an impact on our storage. We are already on 0.5 sample rate, but looks like we need to go even lower, which will degrade developer experience / debugging.

Workaround from our side could be:

• reducing amount of tags on cache entries
• reducing sample rate for traces
• disabling OTel integration for FusionCache
• keep increasing storage everyday

Questions I have to you are:

• can those spans be merged together into one, and attribute fusioncache.operation.key be changed type to array?

[tvardero]

Related to what @EikeSchwass said in #570 (comment):

I would like to piggyback of this and would propose that more control is also given if a span is recorded at all.
For example I don't care so much about "get from cache" (in fact the number of these spans is actively hurting observability), but would still like to see "execute factory".

[jodydonetti]

Hi @tvardero

Overall spans with op get or set cache are more common than even db calls.

Well I would hope so 😅
It means that you are getting from cache way more than you are getting from the db, which is the whole point of caching (at least in a read-heavy scenario, which is the ideal scenario to use caching).

Anyway, read below...

We don't use cache that much, but it already has large impact on the application. We have one cache call for authorization check (on every endpoint), but other than that, we rarely cache anything else in application.

Got it, but that is already 1 call for every single for every endpoint, so it is to be kind of expected.

I can see this:

which means you also enabled traces not just at the "top-level" (e.g.: GetOrSet/TryGet calls) but also at the inner calls level (e.g.: L1 and L2 reads/writes, etc).

Investigating further I see that one GetOrSet operation generates 7 different span (1 parent and 6 child), depending on how many tags were specified for the cache entry. Having more tags generates more spans.

There are 2 different things to consider, for each top-level opration:

• multiple spans for the inner calls (L1/L2/backplane/etc)
• multiple spans for the tagging-related calls (as you probably know, tagging is designed on top of normal entries)

To reduce the spans generated by the first point (inner calls), you can already change the OTEL instrumentation setup, you probably have something like this:

.AddFusionCacheInstrumentation(options =>
{
  // YOU CAN DISABLE THESE TO REDUCE THE AMOUNT OF SPANS EMITTED
  options.IncludeMemoryLevel = true;
  options.IncludeDistributedLevel = true;
  options.IncludeBackplane = true;
})

To reduce the spans generated by the second point (tagging-related stuff) well... you can't do anything right now, BUT I have in my backlog a complete overhaul of the observability part of FusionCache.

This would include stuff, in random order, like:

• better customization for the log records, like options to have them less dense/verbose
• log scopes/bags
• more granular control over the spans being generated
• align to existing caching-related semantic conventions out in the wild (did not exist back when I created the OTEL integration, but they do exist now)
• and most importantly the ability to suppress tagging-related observability (logs, traces, etc) so the tag-related "inner working stuff" can disappear, if desired, to make that transparent (honestly? for me it's a big yeah 🎉)

Now, since I said "in my backlog", you are probably wondering "when?", right 🙂 ?

I'm finishing my Deep Dive course (the second part of this) and it has become something... monumental.

As soon as I will finish it (at most 1 month and it will be out), I'll go back to my backlog, and the first item I'll tackle will be exactly the observability overhaul.

Please hold tight, I'll be on this very soon.

Hope this helps.