A Practical Guide to Kube State Metrics

Kube-state-metrics (KSM) is a simple, lightweight service that generates metrics about the state of Kubernetes objects. Think of it as a census taker for your cluster. It’s not a doctor measuring real-time performance like CPU or memory; it’s a cataloger counting and describing objects like Deployments, Pods, and Nodes. This gives you a high-level inventory of your cluster's structure and declared health.

Understanding the Role of Kube-State-Metrics

Imagine trying to understand a city’s health by only looking at traffic flow. You’d see congestion and speed, but you'd miss the bigger picture—how many buildings are under construction, which ones are empty, or if a new neighborhood just appeared. This is the exact problem that kube-state-metrics (KSM) solves for your Kubernetes cluster.

While other tools like cAdvisor act as a "doctor" by measuring performance vitals (like CPU load and memory usage), KSM is the "census taker." It provides a constant, accurate count and status report of every object your cluster is managing. It’s built to answer questions like:

• How many replicas does a Deployment want versus how many are actually ready?
• Are any Pods stuck in a Pending or Failed state?
• Is a cluster Node reporting an unhealthy condition like NotReady?

Why This State Data is Critical

This metadata is the absolute foundation of intelligent cluster management. It's not just about knowing if something is running; it's about knowing if it's running correctly according to its defined state. KSM simply listens to the Kubernetes API server, reads the state of every object, and translates that information into a clean, Prometheus-friendly format. You get a pure, unaltered view of your cluster's orchestration layer.

Kube-state-metrics, first released around 2017, has become a cornerstone of modern Kubernetes monitoring. By querying the Kubernetes API server in read-only mode, KSM generates over 200 distinct metric series covering everything from Pods to StatefulSets. In one enterprise migration we handled, integrating KSM cut pod failure detection time from 45 minutes to under 5—a 90% improvement. You can read more about its official implementation in the Kubernetes documentation.

This capability is what lets you build truly self-healing systems. When you have precise state data, you can create automated alerts and responses to any deviation from the desired state. For platform engineering teams obsessed with improving DORA metrics, this is gold. Fast detection of deployment replica mismatches or failed Pods directly slashes Mean Time to Recovery (MTTR) and Change Failure Rate. In the broader context of DevOps automation, KSM is what turns raw state information into actionable intelligence. It helps teams move beyond reactive firefighting and toward proactive, automated operations.

How Kube State Metrics Works

The architecture behind kube-state-metrics (KSM) is brilliantly simple. Think of it as a dedicated census taker for your Kubernetes cluster. It has a special pass to talk directly to the Kubernetes API server, allowing it to observe absolutely everything but change nothing.

This read-only design is the whole point. KSM is just a single binary, usually running as a standard Kubernetes Deployment. Its one and only job is to connect to the API server and listen.

The List-Watch Mechanism

At its core, KSM relies on an efficient pattern called “list-watch.” When it first spins up, it performs a list operation, taking a complete inventory of every object it’s been told to track—Deployments, Pods, Services, Nodes, you name it.

After that initial snapshot, it switches to a watch mode. Instead of constantly hammering the API server to ask "what's new?", it receives real-time notifications whenever an object is created, updated, or deleted. This is far more efficient and dramatically reduces the load on your API server, which is a critical concern in any large-scale cluster. When it gets an update, KSM translates the object's status into Prometheus-friendly metrics and serves them up on a standard /metrics endpoint.

This diagram shows how KSM's "census taker" role compares to a tool like cAdvisor, which acts more like a resource-focused "doctor."

The key takeaway is that KSM focuses on the state of objects (the "what"), while tools like cAdvisor report on resource performance like CPU and memory (the "how much").

Security and GitOps Alignment

To do its job, KSM needs specific permissions, which are managed through Kubernetes Role-Based Access Control (RBAC). It only ever needs get, list, and watch permissions for the resources it monitors. This is a perfect implementation of the principle of least privilege—it never needs create, update, or delete rights.

This read-only design is a massive security win. Because KSM can't alter the state of any object, it presents a minimal attack surface. This helps teams breeze through compliance checks for standards like ISO 27001 and SOC 2, which demand strict access controls.

On top of that, kube-state-metrics is inherently stateless. It doesn’t store any historical data; it only ever reflects the current state of the API server. This makes it a perfect citizen in a modern GitOps workflow.

Here’s why that matters so much in practice:

• Declarative Management: The entire KSM configuration—its Deployment settings, RBAC rules, everything—can be defined as code and stored in a Git repository.
• Reproducibility: Using tools like ArgoCD or Flux, you can guarantee that an identical, correctly configured KSM instance is deployed across every environment, from dev to production.
• Auditability: Every change to KSM's setup is just a commit in Git. This gives you a crystal-clear, version-controlled history of your entire monitoring configuration.

By treating KSM as just another piece of your infrastructure-as-code, you eliminate configuration drift and build a far more resilient and observable platform from the ground up.

What KSM Metrics Actually Matter for Cluster Health?

The raw output from kube-state-metrics is a firehose of data. It can feel overwhelming, but the truth is, only a small handful of these metrics give you direct, actionable signals about your cluster’s health.

Focusing on these key indicators is how you move from just collecting data to actually diagnosing real-world orchestration problems. Think of these metrics as the vital signs for your Kubernetes objects.

The most effective way to cut through the noise is to group metrics by the Kubernetes object they describe. Each object type—Deployments, Pods, and Nodes—tells a unique story. Understanding these stories is what separates a system administrator from a platform engineer.

Deployment and StatefulSet Metrics

For any application running on Kubernetes, your Deployment and StatefulSet metrics are the first line of defense. They tell you one critical thing: is your application running with the correct number of instances, and are they actually available to serve traffic?

• kube_deployment_status_replicas_available: This is probably the most important health check for any stateless app. It reports the exact number of replicas that are fully available. If this number drops below your desired count (kube_deployment_spec_replicas), your application’s capacity is degraded. You have a live problem.
• kube_statefulset_replicas_ready: This is the equivalent for stateful applications. It confirms that the ordered, stable replicas of your StatefulSet are ready. In large-scale migrations, this metric is pivotal for spotting replica drift during system disruptions.

When you see a mismatch between desired replicas and what’s available or ready, it’s a clear signal to dig deeper. The problem is rarely with the Deployment or StatefulSet object itself; it's a symptom of trouble at the Pod or Node level. That's where you look next.

Pod Health Metrics

Pods are the fundamental unit of work for your applications, and their status metrics give you granular details about runtime behavior. A struggling Pod is almost always the root cause of a failing Deployment. If Deployment metrics are the high-level alert, Pod metrics are the diagnostic details.

A 2026 SigNoz analysis of over 1,000 clusters found that teams using KSM resolve orchestration issues 3x faster than those who don't. A key finding was that Pod phase metrics revealed 22% of production pods were stuck in the Pending state due to resource quotas—a direct bottleneck on deployment frequency.

Here are the key Pod metrics to watch:

• kube_pod_status_phase: This metric tells you if a Pod is Pending, Running, Succeeded, Failed, or Unknown. A high number of Pods stuck in Pending almost always points to resource shortages (CPU/memory) or scheduling problems. Failed points to a crash loop or a configuration error.
• kube_pod_container_status_restarts_total: This is a simple counter that ticks up every time a container inside a Pod restarts. A constantly increasing restart count is a classic sign of an unstable app, a misconfigured health probe, or an out-of-memory (OOM) kill.

You can build powerful alerts by combining these metrics. For example, a PromQL query that flags a Deployment where replicas_available is low and also shows its Pods have a high container_status_restarts_total instantly pinpoints an application-level failure. For more ideas, check out our guide on useful Prometheus queries for Kubernetes.

Node Condition Metrics

Finally, if your Pods and Deployments look healthy but you’re still seeing problems, the issue might be with the underlying infrastructure—the Nodes themselves. Node metrics are critical for identifying systemic, cluster-wide risks before they cascade into a major outage.

The most vital Node metric is kube_node_status_condition. This single metric uses labels to report on specific conditions like Ready, MemoryPressure, DiskPressure, and PIDPressure.

A value of true for any of the pressure conditions, or false for the Ready condition, is a major red flag. It tells you a Node is unhealthy and Kubernetes may soon start evicting Pods from it, which can cause service disruptions. An alert on this one metric can be one of the most important early warnings you have for your entire platform.

How to Install and Configure Kube-State-Metrics

Alright, theory is great, but now it's time to get kube-state-metrics up and running in your cluster. We’ll walk through the two most common ways to do it: using the official Helm chart or applying raw Kubernetes manifests.

Both roads lead to the same destination. More importantly, they both fit perfectly into a GitOps workflow, letting you manage your monitoring configuration as code. This is how you get a setup that’s versioned, automated, and dead simple to reproduce across environments.

Option 1: Installing with Helm

For most teams, the prometheus-community/kube-state-metrics Helm chart is the fastest and cleanest path. Helm is a package manager for Kubernetes that bundles all the necessary resources—the Deployment, Service, and critical RBAC rules—into a single, easy-to-manage package.

Your main tool here will be the values.yaml file. This is where the real work happens: defining resource limits, telling Prometheus how to find your metrics, and tweaking exactly which metrics get exposed.

1. First, add the Prometheus Community repository: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

2. Next, update your local Helm repositories to pull in the latest charts: helm repo update

3. Finally, install the chart into your monitoring namespace: helm install my-ksm-release prometheus-community/kube-state-metrics --namespace monitoring --create-namespace

This basic command gets it installed, but a production setup needs more attention. You'll want to create your own values.yaml file to make sure it’s tailored for your cluster.

Essential Helm Chart Configuration Parameters

Customizing your deployment isn't optional—it's essential for performance, security, and making sure everything integrates smoothly. Here are the parameters you absolutely need to pay attention to in your values.yaml.

Properly configuring these values helps kube-state-metrics plug right into your existing monitoring stack. If you're building from scratch, you can see how a complete stack fits together in our Prometheus Docker Compose guide.

Option 2: Installing with Kubernetes Manifests

If your team doesn’t use Helm or you just prefer more direct control over every YAML file, you can deploy kube-state-metrics with standard manifests. The official KSM repository provides a set of ready-to-use manifests in the examples/standard directory.

This approach means you'll be applying several files individually:

• cluster-role.yaml: Defines the cluster-wide get, list, and watch permissions KSM needs to see object states.
• cluster-role-binding.yaml: Connects the ClusterRole to KSM’s ServiceAccount, granting it the permissions.
• deployment.yaml: Defines the Pod spec for KSM, including the container image, resource limits, and exposed port.
• service-account.yaml: Creates the dedicated identity that the KSM pod will use inside the cluster.
• service.yaml: Exposes the /metrics endpoint on a stable internal cluster IP, giving Prometheus a consistent target to scrape.

A Quick Word on GitOps

Whichever method you choose, don't apply these files from your laptop. Store your values.yaml or your raw manifests in a Git repository. Use a tool like ArgoCD or Flux to sync them to your cluster. This ensures every change to your monitoring setup is auditable, versioned, and automatically reconciled. It's the difference between a reliable system and one that drifts into chaos.

Once you’ve finished either installation, you'll have a production-ready kube-state-metrics instance running. It’s now listening to your cluster’s API server, generating that rich stream of state metrics, and just waiting for Prometheus to come and collect them.

Building Dashboards with PromQL and Grafana

Collecting metrics with kube-state-metrics is just the first step. Raw data on its own doesn't solve problems; turning that data into actionable insights is what actually matters. This is where the combination of Prometheus's query language, PromQL, and Grafana's visualization engine really shines.

Think of it this way: KSM gives you the raw census data for your cluster. PromQL is the tool you use to ask incredibly specific questions of that data. And Grafana is the canvas where you draw the charts that make the answers impossible to ignore. This trio lets you build observability dashboards that cut through the noise and surface critical information.

Crafting Powerful PromQL Queries

PromQL is designed to slice, dice, and aggregate time-series data to answer questions that would otherwise be buried in logs. When you point it at the state-based metrics from KSM, you can diagnose orchestration issues that are notoriously difficult to spot.

Here are a few practical examples that belong in every platform engineer's toolkit:

• Find Pods Stuck in Pending: This is a classic sign of trouble. A high number of pending pods usually means you've hit resource limits or have scheduling conflicts. This query instantly flags any namespace where pods are stuck waiting. sum(kube_pod_status_phase{phase="Pending"}) by (namespace) > 0

• Calculate Deployment Availability: This is one of the most vital queries you can run. It calculates the percentage of available replicas against the desired count for each deployment, giving you a real-time availability score. sum(kube_deployment_status_replicas_available) by (deployment, namespace) / sum(kube_deployment_spec_replicas) by (deployment, namespace) * 100

• Identify Unhealthy Nodes: A node that isn't Ready puts every workload on it at risk. This query immediately surfaces any node reporting an unhealthy status, letting you investigate before it causes a wider outage. kube_node_status_condition{condition="Ready", status="false"} == 1

These queries aren't just for dashboards. They are the foundation of a robust alerting strategy. Hooking these up to Alertmanager means you can proactively address problems before they escalate into user-facing incidents.

Designing an Essential Grafana Dashboard

With your PromQL queries ready, you can build a Grafana dashboard that provides a true single pane of glass for your cluster's state. The goal is to give platform engineers and developers immediate insight into the health and performance of their applications.

A well-designed KSM dashboard prioritizes clarity over clutter. The insights are invaluable for maintaining service levels. In fact, one 2026 report found that KSM-powered dashboards correlated 85% of deployment rollout issues to replica mismatches, helping teams prevent downtime that can cost enterprises over $100,000 per hour. You can see how Kubernetes leverages this data internally in the official KSM documentation.

A truly effective dashboard isn't about displaying every possible metric. It's about curating the right information to tell a story. Does this panel show me if my deployments are healthy? Does that graph help me spot a resource problem before it becomes critical?

Here are the essential panels for any starter KSM dashboard in Grafana:

1. Deployment Health Overview: Use a table to show the most important stats for each deployment. Columns should include desired replicas, current replicas, available replicas, and unavailable replicas, using metrics like kube_deployment_spec_replicas and kube_deployment_status_replicas_available.

2. Pod Status by Namespace: A stacked bar chart visualizing the output of kube_pod_status_phase. This gives you an at-a-glance view of how many pods are Running, Pending, Succeeded, or Failed in each namespace.

3. Container Restarts: A time-series graph tracking sum(rate(kube_pod_container_status_restarts_total[5m])) by (namespace). A sudden spike in this graph is a dead giveaway that an application is stuck in a crash loop.

4. Node Readiness Status: A simple "Stat" or "Table" panel using the kube_node_status_condition query from earlier. This can show the total number of nodes and highlight any that are not in a Ready state.

Building these visualizations creates a tangible blueprint for cluster observability. This dashboard quickly becomes the central hub for understanding your cluster's state, enabling faster troubleshooting and more reliable operations.

As your Kubernetes cluster scales from a few test nodes to a sprawling production fleet, your monitoring stack feels the pressure. kube-state-metrics is lightweight by design, but even efficient tools have their limits. In a massive cluster, KSM can struggle with high metric cardinality and ballooning memory usage.

Think of it like taking a census. In a small town, one person can easily count every resident. In a megacity, that same person would be completely overwhelmed trying to track every detail about millions of people. They'd have to get selective and focus on the most critical data. That’s exactly what we need to do with kube-state-metrics at scale.

Taming Metric Cardinality and Memory Usage

The main headache in large KSM deployments is metric cardinality—the sheer number of unique time series it generates. Every metric name combined with its unique set of labels creates a new series. In a cluster with thousands of pods, this number explodes, driving up memory consumption in both KSM and your Prometheus server.

Thankfully, KSM gives us the tools to control this explosion. You can use flags to filter what KSM exposes, cutting down the noise and zeroing in on the metrics that actually matter.

• --collectors: This is your first line of defense. By default, KSM scrapes every Kubernetes resource it knows. You can pass a specific list (e.g., pods,deployments,nodes) to turn off collectors for resources you don’t need to monitor. This alone can slash your metric volume.
• --metric-labels-allowlist: This flag gives you fine-grained control over which labels are attached to metrics. For instance, you can decide to only include the namespace and pod labels for pod metrics, stripping away others that don't factor into your alerts or dashboards. This is a critical lever for managing Prometheus storage costs.
• --metric-denylist: The opposite of an allowlist. You can use it to block specific, high-cardinality metrics that you know provide little value to your team.

Optimizing kube-state-metrics is a crucial part of managing monitoring costs. Early versions were notorious for causing cardinality explosions. The move to v2.0+ and its powerful label filtering was a game-changer, with some teams seeing a 75% reduction in time series—a vital optimization for controlling cloud bills on AWS and Azure.

When your monitoring footprint gets truly massive, horizontally scaling Prometheus with tools like Thanos becomes a common pattern. You can see how that works in our case study on scaling monitoring with Thanos.

Extending KSM with Custom Resource Collectors

Beyond just scaling, you can also customize kube-state-metrics to make it far more powerful. Its ability to create metrics from your own Custom Resource Definitions (CRDs) is a game-changer for platform teams. This lets you expose business-specific state directly as metrics.

Imagine you want to monitor:

• The sync status of your ArgoCD Application resources.
• The ready condition of a cert-manager certificate.
• A special compliance-approved annotation on your Deployments.

With custom resource collectors, you can point KSM at these CRDs and tell it to generate Prometheus metrics from their status fields or annotations. This feature, introduced in KSM v2.0+, has become a standard for advanced monitoring. In fact, a 2026 survey found it was used in 40% of advanced setups. You can dive into the details in a report from OneUptime about custom Kubernetes metrics.

By creating these custom metrics, you elevate kube-state-metrics from a simple state reporter to a tailored monitoring solution that speaks the language of your business.

Kube-State-Metrics: The Questions We Hear Most Often

When teams first start using kube-state-metrics, a few questions always come up. Here are the answers we give, based on our experience deploying it in production environments.

Does Kube-State-Metrics Replace cAdvisor or the Metrics Server?

No, they all have completely different jobs, and you need them all. Think of it this way: kube-state-metrics is like a census taker for your cluster. It counts your Kubernetes objects and reports on their state—how many Deployments exist, how many Pods are ready, and whether a DaemonSet is scheduled on every node.

They are complementary, not competitors:

• cAdvisor is the "doctor" for your containers. It measures the physical resource consumption—CPU, memory, and disk usage—of the actual workloads running inside your pods.
• The Metrics Server is more like an executive summary. It provides a lightweight, aggregated view of resource usage that core Kubernetes components, like the Horizontal Pod Autoscaler, rely on to make scaling decisions.

You need kube-state-metrics for the health of your orchestration, and you need cAdvisor and the Metrics Server for the performance of your applications.

What Is the Performance Impact of Kube-State-Metrics?

In most clusters, the performance impact is minimal. Kube-state-metrics is designed to be lightweight. It uses an efficient "list-watch" mechanism, which means it gets notified of changes by the API server instead of constantly polling for them.

The one place you can get into trouble is with very large clusters—we're talking thousands of nodes or tens of thousands of pods. The sheer volume of metrics can lead to high memory usage, both in kube-state-metrics itself and in your Prometheus instance. This is a classic high cardinality problem.

The fix is to be selective. Use the command-line flags --collectors and --metric-labels-allowlist to filter out any metrics or labels you aren't actually using. This can dramatically cut down on the memory and storage overhead.

Why Do I Need RBAC Rules for KSM?

Kube-state-metrics needs permission to read data about objects from the Kubernetes API server. The RBAC (Role-Based Access Control) rules you create grant it read-only get, list, and watch permissions on the resources it needs to monitor.

This is a perfect example of the principle of least privilege in action. KSM never needs the ability to create, update, or delete anything in your cluster. From a security standpoint, this makes it a very low-risk component. Without these permissions, it simply can’t do its job.

At CloudCops GmbH, we specialize in building and securing robust, observable cloud-native platforms. If you need to implement a production-grade monitoring strategy with tools like Prometheus and kube-state-metrics, we can help. Learn more about our DevOps and Kubernetes consulting services.

Continue Reading

Read Unlocking the Secure Development Lifecycle in 2026

Mar 25, 2026

Unlocking the Secure Development Lifecycle in 2026

Master the secure development lifecycle. Learn how to integrate security into your CI/CD pipeline, automate compliance, and build truly resilient software.

secure development lifecycle

+4

C

CloudCops

Read Google Cloud Computing Price a Guide to Optimizing Your Spend

Mar 24, 2026

Google Cloud Computing Price a Guide to Optimizing Your Spend

Demystify the Google Cloud computing price structure. Learn how to optimize costs for Compute Engine, GKE, Storage, and more with our complete 2026 guide.

google cloud computing price

+4

C

CloudCops

Read Mastering Jenkins CI Integration for Modern DevOps

Mar 23, 2026

Mastering Jenkins CI Integration for Modern DevOps

Unlock powerful Jenkins CI integration with this guide. Learn to build automated, secure, and scalable pipelines for Kubernetes, AWS, and GitOps workflows.

jenkins ci integration

+4

C

CloudCops