When a metric has so many possible values on its labels that the monitoring stack needs to create a new time series for every permutation, we call that high cardinality. It leads to increased resource consumption, decreased performance and increased cost. And it's almost never solved by simply adding resources.
You have to draw the line between what should be monitored via metrics, versus what can be captured elsewhere such as logs and distributed tracing, and ensure this happens prior to the data entering your monitoring platform. As discussed below, this becomes critically important once you start aggregating metrics from a large fleet of equipment, for example in datacentres, construction sites and business parks. This is central to the solution we provide with PODVIEW.
PODTECH Keep Infrastructure Monitoring Dependable
PODTECH develops monitoring solutions to support the datacentre and smart-building industries.
VisitTL;DR
- Time series cardinality equals the number of unique time series created. This is the number of permutations of a given metric's label set.
- Cardinality compounds quickly if a highly-variable label is added to an existing metric.
- Typical high-variance labels include users, requests, containers and ephemeral IPs.
- Move highly variable detail to logs and other sources where appropriate, and restrict the range of values observed for certain labels.
- Audit current time-series counts per metric and label, and only strip away those that don't feed alerts, dashboards and root cause analysis.
Table of Contents
- What cardinality means in monitoring
- Why high cardinality hurts
- What makes cardinality grow
- How to check your cardinality
- How to reduce cardinality without losing diagnostic detail
- High cardinality beyond servers: buildings and sites
- A cardinality review checklist
- Where PODTECH fits
- Our perspective
- FAQ
- Sources
What cardinality means in monitoring
As with any other context, cardinality refers to the size of a set of items. Here we're concerned with sets of time series. A single time series consists of successive measurements made over time, for example from an individual data point. If we define a metric such as inlet_air_temp for a set of server racks, and add a series of additional labels such as site, hall, rack and temp_sensor, we will see multiple time series generated. The cardinality of this metric therefore corresponds to the number of unique time series. Aggregating across all defined metrics, we get the overall cardinality of our environment.
Time-series platforms will create some form of record per time series, typically backed by a database index and associated in-memory data structures. Hence the cardinality of the environment is generally the key factor impacting platform requirements and spend.
Of course, it is also helpful to think about the cardinality of each individual label, as the total number of series will be bounded by this, multiplied together, subject to the permutations seen in practice.
Example
Suppose we want to monitor the power usage of each datacentre rack across a number of locations:
| Label | # of unique values |
|---|---|
| Site | 6 |
| Hall | 4 × site |
| Rack | 120 × hall |
| Phase | 3 |
We would expect to generate roughly 8,640 time series from this metric. Adding further labels may be necessary, but suppose we wanted to know which version of the rack's controller was installed, and added a label for controller_firmware. If there were 25 values, we'd now see roughly 216k series. Finally, if another group decided to add a session ID to their metrics, they may well see an effectively unbounded increase. There is likely nothing wrong with the systems generating these metrics, but your monitoring infrastructure will be significantly affected.
Why high cardinality hurts
There are several consequences, broadly as follows:
Increased resource demand. Palo Alto Networks explain this well, but in short, high cardinality requires greater compute and memory in both the data collectors and TSDB. Obviously this scales with the number of active series.
Degraded query performance. Any query that spans a large number of series will incur higher latencies. Because this will generally creep up over time, you may only discover the impact when using your platform during a fault.
Higher cost. Many cloud-based monitoring solutions will charge directly based on the number of active series and or custom-defined metrics. With an on-premise offering, you will pay for bigger machines and greater storage. It is easy for a single poorly-chosen label to become a significant cost driver.
Lastly, and perhaps most importantly, if the cardinality of your environment causes the monitoring system to fail, you have removed your ability to observe issues in the assets that you most care about.
What makes cardinality grow
As highlighted above, there are a handful of obvious candidates that tend to drive the problem. User ID, Request ID, Container Name and Temporary IP are all commonly-cited examples, as is the natural consequence of running large numbers of ephemeral Kubernetes Pods, and metrics emitted by downstream third parties. All of these will be applicable in other contexts too, as we'll see below.
- Unique per entity key. Some form of unique key associated with a person, connection, business event, work order or access card read. These will be unique and therefore generate a new series.
- Transient resources. Pods and containers will typically receive a new identifier when redeployed. Any prior series will cease to accumulate samples but remain in the overall tally until purged. See our Kubernetes page for more on how Kubernetes handles this.
- Source and dest IPs and ports. There are many combinations of source and dest IPs, and the pool of transient ports is very large.
- Arbitrary strings. For example complete HTTP path including any IDs, exception text, GET parameters and similar values.
- All label combinations for a given histogram. Since there is a series for each bin in a histogram, adding multiple further dimensions will result in even higher cardinality.
- Exporter-defined keys. Many exporters and other sources will append all applicable labels to their output.
- Large numbers of tagged assets. Sensor deployments may comprise tens of thousands of individual units, each reporting multiple data points. While high-cardinality in principle, it is often unavoidable.
Be careful to distinguish these final two cases. Identifiers that are inherently unbounded are almost always anti-patterns. A known quantity of physical devices simply requires adequate capacity.
Assessing your current cardinality
As with all aspects of managing a time-series DB, you should know the current state of things before attempting to change it. Although the syntax varies, you will generally want to:
- Establish the existing number of series, and examine how it has changed over the previous 30 to 90 days. Be suspicious if the increase does not correlate with added entities.
- Determine which metrics contribute most to the aggregate. Typically, a small proportion of metrics will make up the bulk of your series.
- Inspect the cardinality contributed by each dimension for those problem metrics. One will usually stand out.
- Measure the volume of new series generated per unit time. This helps to identify sources of unbounded identifiers.
- Identify who owns each problematic metric, whether that is a responsible team, exporter or third party. You will not see reductions unless you can contact someone.
- Establish whether that dimension is used in any monitoring artifacts. Where the answer is no, you can safely drop it.
Perform this review regularly, for example monthly, and whenever significant additions to your environment are made. You will find any improvements erode after roughly three months.
Reducing cardinality
There will come a point where further reductions have diminishing returns. The target is simply to ensure you have enough granularity to address your key business needs. Proceed as follows:
1. Clarify the purpose of metrics
In general, we want to track some property of interest numerically. This could be frequency, cumulative total, distribution, resource consumption and so on. While it can be useful to include certain keys to aid triage, Palo Alto Networks believe this is better suited to the log and distributed tracing space. In other words: if the desired answer to a query is numeric, it belongs in a metric; if it requires an explicit match against, for example, an individual HTTP call, then log it.
2. Eliminate unused labels
If no piece of monitoring consumes a particular label, there is little justification for retaining it. The most effective approach is to prevent the relevant series from being emitted in the first place.
3. Generalise
Most of the time you won't require the precise value of a given key. Consider collapsing an HTTP endpoint to a URI pattern, an IP to its network segment or data centre, a user to their customer classification, a pod to its parent service, or an exception string to its error code. You'll retain sufficient context to take action.
4. Pre-summarise
If your users predominantly want to see a given metric aggregated across some other axis, consider persisting only that summary. Some TSDBs offer mechanisms to enforce this, for example recording and aggregation rules.
5. Limit exposure
Impose a maximum number of series and or label values per source at point of ingestion. Better to fail the scrape and fire an incident than suffer memory exhaustion on your monitoring host. Coupled with an agreed convention for metric definition, this will go some way to controlling future growth.
6. Retain critical granularity
There will always be cases where having higher granularity is justified. Monitoring an individual device's readings in a datacentre is the most obvious example: you want to know if that particular chiller, gen-set or water leak detection loop is going to fail. How do you know if you have too much detail? Ask yourself: would anyone, human or otherwise, behave differently based on this? If yes, keep it. If no, remove it unless there's another compelling business case for retaining it.
When should you retain high cardinality?
| Scenario | Justified? | Why / Alternative |
|---|---|---|
| One time series per entity | Yes | Limited number of entities |
| Time series per HTTP request, user session etc | No | Potentially infinite |
| Logs, distributed tracing | No | Entity varies per release |
| Group logs/traces by logical job | N/A | See above |
| Source and dest IP(s) | Only some cases | Preferably aggregated by network segment |
| Breakdown by customer | Selected | Aggregate remainder |
Other large-scale entities
Any time you're aggregating readings from multiple devices onto a common monitoring backend, you'll encounter these considerations. A typical build site may have dozens or hundreds of active machines, worker devices and various site-wide environmental instrumentation. You'll likely want to monitor each piece of plant in detail, but also get a macro-picture of the overall risk profile at site level. We discuss the benefits of combining high-level metrics and device-level telemetry in IoT in asset management. Further reading on safety at build sites using IoT includes our work on connected-site safety.
Similarly, the Building Management System, or BMS, will output time series at the level of individual points such as pumps, fans and M&V instruments. Attempting to simply tag these arbitrarily and push them through to a metrics database is likely to lead to a costly production deployment. Define your fixed levels of hierarchy, and ensure they map to a limited set of values. All additional metadata can then be stored against each time series without forming part of its key. See our definition of the relevant acronyms in the BMS glossary.
Questions to ask yourself
Before introducing a new metric, exporter and third-party feed:
- Do I know what values each dimension can take, and can this be expected to remain true as the business scales?
- Do I have IDs, points-in-time, strings, locations or cryptographic hashes embedded as dimensions?
- Are there dashboards, alerts or scheduled jobs dependent on me being able to query by each of these?
- Am I exposing histograms, and is my choice of number of buckets multiplied by other dimensions appropriate?
- Can I identify a single point of accountability, and be alerted if I breach the acceptable number of time series?
- Is a similar cap applied upstream, at source?
- Where is the relevant extra information best captured?
By setting a series quota per squad or application, and revisiting this regularly, you can make cardinality a known factor.
How PODTECH can help
These issues are rarely solved by tweaking an ad-hoc solution in isolation. Rather, they stem from wider design choices made around instrumenting, gathering and consolidating information. This is our area of expertise. PODTECH develops bespoke technology platforms to support critical technical assets, and has successfully implemented more than 250 such solutions. Datacentre monitoring and instrumentation is a key focus within these offerings.
PODVIEW is our offering for live datacentre performance and diagnostic analytics, comprising three complementary products: a monitoring suite, range of fixed and RF-based sensors and a control interface. Should your current arrangement be reaching its natural end-of-life, our Legacy Modernisation team can revise the telemetry pipeline to incorporate inputs from your BMS, PMS and NMS. And via our Machine Learning arm, we can develop advanced diagnostic and predictive functions from suitably-normalised inputs. See our article on infrastructure observability for further discussion.
Get in touch with PODTECH to explore developing a tailored telemetry framework for your operations.
Alternatively, we can engage directly with your in-house engineering function to augment your capabilities. In either event, our offering is backed by a 99.9% availability guarantee.
Our view on the topic
As noted above, we commonly find that the impetus behind an inquiry into excessive cardinality is a desire to capture as much raw information as possible. Storage is relatively inexpensive, and more data will only improve the quality of insights until it doesn't. At which point, the time to diagnose the root cause can be long.
Our approach is to start with the required business outcomes, in other words what decisions need to be supported by the monitoring system. For a datacentre, this generally maps to a set of clearly-defined, hierarchically-structured levels across which time series can be exposed. Where there is a risk of failure, we will retain the maximum resolution of data possible, and direct all non-repeatable details to suitable alternative stores.
Reliable monitoring of critical facilities requires a robust software foundation to continue operating during periods of elevated load, and to communicate findings concisely. As such, we tend to scope telemetry, consolidation and insight-generation holistically.
Harry
FAQ
What constitutes high cardinality?
It's down to judgement. If the possible range of values is very large, or potentially infinite compared with the capacity of your platform, then it will be considered high cardinality. Typical candidates include things like users, requests and containers. A label with, for example, 12 discrete values would be low.
What is the cardinality of my metric?
You need to count how many unique series, meaning combinations of label values, exist under that metric. A first approximation can be made by multiplying the unique value counts per label. For our previous example, this would be 6 × 4 × 120 × 3 = 8,640 series. Your monitoring backend should also report this.
Can't high cardinality be useful?
If your business operates a limited set of key devices, it's highly desirable to track each one as a separate time series. High cardinality becomes problematic only if there's an open-ended or superfluous dimension.
Where does high cardinality fit?
Clearly not in metrics. You want to capture key high-level patterns in metrics, but log or trace level details for each device or event.
Sources
- What Is High Cardinality in Observability? (Palo Alto Networks Cyberpedia)
- IoT in asset management: boost efficiency by 30%