--- title: "Configuration steps for Kubernetes Monitoring with Helm chart | Grafana Cloud documentation" description: "Steps to configure Kubernetes Monitoring with Helm chart" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Configuration steps for Kubernetes Monitoring with Helm chart On the main navigation of Grafana Cloud, navigate to the **Cluster configuration** tab under Kubernetes Monitoring. To configure with the Helm chart wizard, the wizard guides you through the following steps: 1. Backend and distribution: Install the required recording and alert rules, and choose your Cluster’s platform. 2. Access token: Provide the telemetry collector with an access policy token so you can send metrics and logs to Grafana Cloud. 3. Choose setup: Choose a preconfigured deployment of the Alloy collector that meets your Cluster’s scaling and resilience needs. 4. Monitoring: - Choose the telemetry you want to collect for metrics, logs, and events. - Decide whether to have services running in your Cluster automatically discovered so you can instrument them from Grafana Cloud. 5. Deployment: - Enter a name for your cluster and the namespace where the Alloy collector should be deployed. - Copy and paste the Helm command into your terminal. The wizard turns these choices into Helm values and a deployment command. To learn how the chart organizes features, collectors, presets, and destinations, refer to [Helm chart overview](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/). ## Before you begin Make sure you have met the prerequisites required for these configuration steps. > Note > > Ensure that you are familiar with the [components installed by the Helm chart](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/#packages-installed-with-helm-chart) and how they relate to switching on or off the configuration choices available. To deploy Kubernetes Monitoring with the Helm chart, you need: - A Grafana Cloud account - The Admin role to install alerts - To have completed the [activation of the app](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/activate/) - A Kubernetes Cluster, environment, or fleet you want to monitor - The `kubectl` and [Helm](https://helm.sh/) command-line tools - To check version compatibility for the features you plan to enable: - kube-state-metrics uses `client-go` to communicate with Clusters. For Kubernetes client-go version compatibility, refer to [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics#versioning). - Cost metrics with OpenCost require [Kubernetes 1.8 or later](https://github.com/opencost/opencost#getting-started). - Visualizations for storage require [Helm chart](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring) [release](https://github.com/grafana/k8s-monitoring-helm/releases) v1.5.1 or later. ## Backend and distribution Install the required recording and alert rules into your Grafana Cloud stack, and choose the platform for your Cluster. - A [recording rule](/docs/grafana/latest/alerting/fundamentals/alert-rules/#recording-rules) pre-computes query results and saves them as new metrics. Dashboards and alert rules then read the saved results instead of running expensive queries each time, which makes queries and alert evaluation faster. Recording rules add new time series to your stack rather than reducing existing ones. The workload views in Kubernetes Monitoring read from the metrics that the recording rules produce. - An [alert rule](/docs/grafana/latest/alerting/fundamentals/alert-rules/) defines the conditions that fire an alert. > Note > > **Backend Installation** installs rules that are *required* for Kubernetes Monitoring to function properly. > > If you aren’t seeing workload data, the most likely cause is that the recording rules and alert rules haven’t been installed. 1. Under **Backend Installation**, click **Install** to install the required, preconfigured alert rules and recording rules to your Grafana Cloud stack. After they’re installed, the status shows **Alert rules and recording rules installed**. 2. Under **Select your Kubernetes platform**, choose your platform: - **Kubernetes**, for EKS on EC2, GKE, and similar distributions - **Azure AKS** - **GKE Autopilot** - **EKS on Fargate** - **IBM Cloud** - **OpenShift** 3. Click **Next**. ## Access token Choose how the Helm chart authenticates to Grafana Cloud. [Grafana Alloy](/docs/grafana-cloud/send-data/alloy/), the telemetry collector that the Helm chart deploys on your Cluster, needs an [access policy token](/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) to send metrics and logs to Grafana Cloud. You can supply that token in one of three ways: create a new access policy token, use an existing token, or point the chart at a Kubernetes Secret in your Cluster that already holds a token. To create a new token: 1. Click **Create a new token**. 2. In the **Access policy token name** box, enter a name. This name helps you view and revoke the token in Grafana.com later. 3. In the **Expiration date** box, select or enter an expiration in days, or keep **No expiry**. The token uses the predefined scopes `metrics:read` and `set:alloy-data-write`. A scope defines which permissions a token has. For more information, refer to [Scopes](/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/#scopes). 4. Click **Create token**. To use an existing token: 1. Click **Use an existing token**. 2. Paste your token into the box. To use a stored Kubernetes Secret: 1. Click **Use a stored Kubernetes Secret**. 2. In the **Secret name** box, enter the name of the Secret that already exists in your Cluster. 3. In the **Namespace** box, keep the installation namespace or enter the namespace that holds the Secret. 4. Optionally, expand the advanced options to change the **Username key** and **Password key** if your Secret doesn’t use the default keys `username` and `password`. The Secret must hold your Grafana Cloud instance ID as the username and an access policy token as the password, with the `metrics:read` and `set:alloy-data-write` scopes. To find your instance ID, refer to [Find instance endpoints](/docs/grafana-cloud/security-and-account-management/cloud-stacks/#find-instance-endpoints). The configuration page provides a `kubectl` command that includes your instance ID, which you can copy to create the Secret. Because the chart reads the token from the Secret at runtime, your token isn’t stored in Grafana and doesn’t appear in the generated Helm command. 5. Click **Next**. ## Choose setup On the **Choose setup** step at the **Choose your Alloy deployment** screen, select a preconfigured deployment of the Alloy collector based on the scale and resilience your Cluster needs. A [collector](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/#collector) is an instance of Grafana Alloy that gathers telemetry from your Cluster and sends it to Grafana Cloud. In the deployment options offered by the wizard, collectors run on every Node (a DaemonSet). Every option deploys collectors. - A *clustered backend* is a second Alloy workload (a StatefulSet) that runs on your own Nodes and collects data that exists once per Cluster rather than per Node, such as Cluster events and state metrics. (It’s unrelated to the **Backend Installation** in the first step, which installs recording and alert rules into your Grafana Cloud stack.) - Autoscaling lets a deployment handle more load without a reinstall. Buffering to persistent storage (a PVC) lets the backend hold data on disk so it survives restarts. Use the following table to determine whether to choose the **Scalable**, **Standard**, or **Lightweight** Alloy deployment. Expand table | Feature | Scalable - High performance (Recommended) | Standard - Balanced cluster setup | Lightweight - Quick start | |------------------------|-------------------------------------------------------------------------------------------|--------------------------------------------------------------------|-------------------------------------------------------------------| | Best for | Large-scale production environments, high-traffic systems, mission-critical services | Most production workloads, growing teams, medium to large Clusters | Smaller Clusters, early stage environments, cost-conscious setups | | Topology | Per-Node collectors plus a clustered backend | Per-Node collectors plus a clustered backend | Per-Node collectors only | | Autoscaling | Yes. Collectors scale vertically, and the backend scales horizontally, as load increases. | No | No | | Data buffering | Persistent storage (PVC) | Limited, no persistent buffering | None | | Traffic spikes | Designed for high and burst traffic | Moderate workloads supported | Limited, may drop or miss data under load | | Operational robustness | High availability, fault-tolerant | Stable for most production use cases | Basic, best-effort monitoring | Click **Next**. ## Monitoring On the **Monitoring** step at **Remote monitoring and service discovery**, choose the telemetry that Kubernetes Monitoring collects from your Cluster, and whether to discover the services running in it. Fleet Management lets you manage Alloy collectors from Grafana Cloud. The [Deployment step](#deployment)’s generated deployment code connects your collectors to Fleet Management automatically, so you can change their remote settings without a Helm upgrade. 1. Choose the telemetry to collect for Kubernetes monitoring. **Enable Kubernetes monitoring** is required, so you can’t deselect it, but you can choose what it collects. Click the menu icon next to **Enable Kubernetes monitoring** and select **Advanced metrics tuning** or **Advanced logs and events tuning**. Switch each type of telemetry on or off, then click **Save**. For details on each option, refer to [Metrics](#metrics) and [Logs and events](#logs-and-events). You can change these choices later, but how depends on the type. Metrics settings apply locally on your Cluster, so changing them means updating your Helm values and redeploying the chart. You manage logs and events settings through [Fleet Management](/docs/grafana-cloud/send-data/fleet-management/), so you can change them without redeploying. 2. Decide whether to select **Auto-discover services on my cluster**. This option finds the services running in your Cluster so you can instrument them from Grafana Cloud. Instrumentation is what makes a service send application telemetry. You can add instrumentation code to the service yourself, or instrument it without code changes through Grafana Beyla. Either way, Grafana Cloud has to know a service exists before it can offer to instrument it. That’s what auto-discovery does. Auto-discovery feeds [Instrumentation Hub](/docs/grafana-cloud/send-data/inst-hub-setup/), a Grafana Cloud tool in public preview. Instrumentation Hub lists the discovered services and gives you one-click instrumentation for each one. Select **Auto-discover services on my cluster** if you want to instrument services from Grafana Cloud. The wizard creates a [configuration pipeline](/docs/grafana-cloud/send-data/fleet-management/set-up/configuration-pipelines/) in Fleet Management. Instrumentation Hub uses that pipeline to discover services. Discovery alone collects no application telemetry. Telemetry starts only after you instrument a service, and discovery doesn’t override instrumentation you already have. Skip this option if you don’t plan to instrument services from Grafana Cloud. Click the menu icon next to **Auto-discover services on my cluster**, select **Managed discovery pipeline settings**, then choose **Not recommended**. The wizard doesn’t create a pipeline, Instrumentation Hub won’t list services from your Cluster, and one-click instrumentation isn’t available. You can still [configure instrumentation in your Helm values](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/customize-helm-chart/#instrumentation-for-applications) at any time. If a discovery pipeline from an earlier setup already exists, choosing **Not recommended** leaves it in place. To remove it, delete it in Fleet Management. > Note > > These options are independent of each other. For example, disabling cost or energy metrics does not disable any other option. Refer to additional information for each option by following the links and review [Manage your Kubernetes configuration](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/#cluster-events). ### Metrics To choose which metrics are collected, click the menu icon next to **Enable Kubernetes monitoring** and select **Advanced metrics tuning**. Metrics run directly on your Cluster. These settings are configured locally and aren’t managed through Grafana Cloud. Switch these metrics options on or off, then click **Save**. - **Cluster metrics**: Collects CPU, memory, network, and storage metrics from Cluster Nodes and Pods. What is collectedWhy enable? What is collected Why enable? Node resource usage; metrics about Pod health; Persistent volume usage; and Deployment, StatefulSet, and DaemonSet status. Essential for Cluster health monitoring; powers the Kubernetes Overview dashboard; tracks resource utilization and capacity planning; and detects Pod crashes, OOM kills, and so on. - [**Cost metrics**](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/) (`costMetrics`): Collects resource costs per namespace and workload to identify expensive Pods and optimize spending. Uses [OpenCost](https://opencost.io/). What is collectedWhy enable?Private registry settings What is collected Why enable? Private registry settings Resource cost attribution by namespace, workload, and Pod; CPU and memory cost breakdowns; Cloud provider pricing data integration; and cost efficiency metrics. Shows cost data; helps identify expensive workloads; enables FinOps and cost optimization; and tracks spending trends over time. If your environment pulls container images from a private registry, you must configure OpenCost’s image source after you deploy. Without this setting, OpenCost Pods become stuck in ImagePullBackOff and no cost metrics are collected. Refer to [Configure OpenCost for a private image registry](#configure-opencost-for-a-private-image-registry). - [**Energy metrics**](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/optimize-resource-usage/monitor-energy-use/) (`hostMetrics.energyMetrics`): Measures power consumption per Node and workload for carbon footprint tracking. Uses [Kepler](https://sustainable-computing.io/). - [**Pod & Service metrics by annotation**](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/annotation-autodiscovery/default/README.md): Scrapes metrics from Pods and Services that have the `k8s.grafana.com/scrape` annotation. What does this do?Why enable?Why disable? What does this do? Why enable? Why disable? Automatically discovers running pods and services; dynamically scrapes Prometheus metrics from annotated Pods; detects new workloads without manual configuration; and uses Kubernetes annotations to find metrics endpoints. Enable for: zero-configuration metrics collection; automatically monitor new applications as they deploy; support microservices architectures with dynamic scaling; and find application-specific metrics beyond system metrics. Disable when: You want to explicitly define every scrape target; you’re concerned about discovering unintended metrics endpoints; your cluster has very strict network policies. - [**Prometheus Operator objects**](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/prometheus-operator-objects/default/README.md): Use this option if Prometheus Operator is already running in your Cluster and you have ServiceMonitor, PodMonitor, or Probe objects. It adds those objects as scrape targets. What does this do?Why enable?Why disable? What does this do? Why enable? Why disable? Discovers and monitors Prometheus Operator custom resource definitions (CRDs). ServiceMonitor defines how to scrape metrics from Kubernetes services, PodMonitor defines how to scrape metrics from Pods, and Probe defines blackbox probing of endpoints. Enable when: you’re already using Prometheus Operator in your Cluster; you have existing ServiceMonitor/PodMonitor definitions; you want to use your existing Prometheus configurations; you want to enable migration from Prometheus Operator to Grafana Alloy. Disable when: You’re not using Prometheus Operator Objects; you prefer using the native configuration of Grafana Alloy. ### Logs and Events To choose which logs and events are collected, click the menu icon next to **Enable Kubernetes monitoring** and select **Advanced logs and events tuning**. These options are deployed and controlled through Grafana Cloud, so you can update and scale instrumentation without changing your Cluster configuration. Switch these logs and events options on or off, then click **Save**. - [**Cluster events**](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/#cluster-events): Collects Kubernetes events, including Pod failures, scheduling events, and resource state changes. What is collectedWhy enable? What is collected Why enable? Generated from: the scheduler (assigning Pods to Nodes); `kubelet` (managing Pods on Nodes); the controller manager (handling scaling, deployments, and so on); the API server (processing requests). Troubleshooting scheduling & deployment issues; see why a Pod isn’t starting (no Nodes with enough memory); tracking resource lifecycle changes (Pod created → scheduled → pulled → started → ready → terminated); detecting transient or recurring failures (repeated image pull errors, failed probes, or node taints); auditing Cluster activity (to identify which controller or user triggered changes). - [**Node logs**](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/#node-logs): Collects `kubelet` and system logs from worker Nodes running systemd, including [journald](https://www.freedesktop.org/software/systemd/man/latest/systemd-journald.service.html) logs. What is collectedWhy enable? What is collected Why enable? Logs from: the operating system (kernel logs, systemd, network drivers); Kubernetes node agents (like `kubelet`); container runtimes (Docker, containerd); system daemons (journald, syslog). Diagnosing node instability (memory exhaustion, CPU throttling, or disk space issues); debugging scheduling or startup failures (when Pods can’t start, the issue may be at the Node level); investigating network or storage problems; determining driver or volume mount failures; auditing system changes (Node reboots, `kubelet` restarts, or OS updates). - [**Pod logs**](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/#pod-logs): Collects application logs from running Pods. Available as `podLogsViaLoki` (Loki pipeline) or `podLogsViaOpenTelemetry` (native OTLP format, the OpenTelemetry protocol) in the Helm chart values. What is collectedWhy enable? What is collected Why enable? Standard output (stdout) and standard error (stderr) Pod logs from the processes in the container, such as initialization messages, API request logs, warnings or errors, and application-specific information. Enable for: debugging issues, such as when an application crashes or behaves unexpectedly, logs reveal what went wrong; monitoring behavior, such as tracking normal operational messages (startup confirmation, API requests, or job completion); auditing events, to view logs that show what actions were taken by your app or scripts running inside containers; gaining performance insight, such as tracing slow operations or bottlenecks using timestamps and log levels. - [**Kubernetes manifests**](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/troubleshooting/#investigate-deployment-changes) (`kubernetesManifests`): Collects Kubernetes resource manifests and changes as logs, using [`k8s-manifest-tail`](https://github.com/grafana/k8s-manifest-tail). Provides the data behind the **View Manifest** and **Changes** tabs on workloads and Pods. What is collectedWhy enable? What is collected Why enable? Full resource manifests for Pods and workloads; creation, update, and deletion events for those resources; and the version history that powers the manifest diff viewer. When switched, you can compare a manifest against previous versions in a side-by-side diff; on Deployment detail page of `k8s-manifest-tail`, you can view a highlight on the graphs in the Optimization section to see if a deployment change correlates with a change in behavior. After you finish choosing the telemetry, click **Next**. ## Deployment 1. In the **Cluster name** box, enter a unique, human-readable name for this Cluster. Use only lowercase letters, numbers, and hyphens, start with a letter, and end with a letter or number (up to 63 characters). This name appears in dashboards, alerts, and Cluster selectors. 2. In the **Namespace** box, enter the namespace where you want to install Grafana Alloy. 3. Optionally, under **Custom labels**, add key and value labels, also known as attributes, to help you search for, identify, and deploy pipelines in Fleet Management. Click **Add more** to add another label. These labels are added to every collector’s `remoteConfig.extraAttributes` map in the generated deployment code. The generated deployment code registers your collectors with [Fleet Management](/docs/grafana-cloud/send-data/fleet-management/set-up/onboard-collectors/kubernetes/) automatically, so you can manage them from Grafana Cloud and change the logs and events settings later without a Helm upgrade. 4. If you don’t already have the Grafana Helm charts repository, add it: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm repo add grafana https://grafana.github.io/helm-charts ``` If you already have it, update it: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm repo update ``` 5. Under **Deployment code to copy**, copy the generated `helm upgrade --install` command and run it in your terminal. Make sure your `kubectl` context points to the Cluster you want to monitor. After the chart deploys and your Cluster starts sending `kube-state-metrics` data, billing begins based on your [active host hours and active container hours](/docs/grafana-cloud/platform/pricing-and-usage/kubernetes-monitoring/#definitions). ### Verify your installation Confirm that the collectors are running and data is arriving in Grafana Cloud: 1. Click **Test connection** to verify that the collectors install and send data to Grafana Cloud. 2. Click **See cluster status** to view the status of data collection. Data appears as the collectors start scraping and sending it to Grafana Cloud. This view shows the health of the different sources of metrics, Pod logs, and Cluster events, as well as any applicable version numbers. This completes the configuration wizard. ## Manage your installation After you complete the wizard, use the following tasks to adjust, troubleshoot, and extend your installation. ### Configure OpenCost for a private image registry If you enabled cost metrics and your environment pulls container images from a private registry, configure OpenCost’s image source. Without this setting, OpenCost Pods become stuck in ImagePullBackOff and no cost metrics are collected. The `helm upgrade --install` command that the wizard generated contains your Helm values. If you don’t have a `values.yaml` file, retrieve your current values first. Refer to [Retrieve Helm values](#retrieve-helm-values). 1. Add this to your `values.yaml` file: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml telemetryServices: opencost: deploy: true opencost: exporter: image: registry: my.private.registry.example.com ``` 2. Deploy or upgrade the chart: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring \ --values values.yaml ``` For other private registry options, refer to [Private image registry](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/customize-helm-chart/#private-image-registry). ### Troubleshoot Refer to [Troubleshooting](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/troubleshooting/) for any issues that occur after configuration. ### Optionally install any integrations You can use Grafana integrations to monitor the health and status of services and applications running in your Kubernetes clusters. To install a Kubernetes integration to begin scraping metrics: 1. From the main menu, navigate to **Connections**, and filter for Kubernetes. 2. Select the integration for the service you want to monitor. 3. Follow the instructions on the screen to copy and use the configuration snippet and install the integration. 4. After installing an integration, redeploy the configuration using the method you originally used. ### Retrieve Helm values If you installed Kubernetes Monitoring with the Helm CLI, you can retrieve the values for your configuration by using the [`helm get values` command](https://helm.sh/docs/helm/helm_get_values/).