Monitor Grafana Mimir on Grafana Labs

Monitor the health of your system

Mon, 31 Mar 2025 14:38:03 +0000

Monitor the health of your system

You can monitor Grafana Mimir or Grafana Enterprise Metrics by collecting metrics and logs from a Mimir or GEM instance that’s running on a Kubernetes cluster. This process is called metamonitoring.

As part of metamonitoring, you can create dashboards and receive alerts about the metrics and logs collected from Mimir. To set up these dashboards and alerts, refer to Installing Grafana Mimir dashboards and alerts.

Configure the Grafana Agent operator via the Helm chart

Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

In the Helm chart, you can configure where to send metrics and logs. You can send metrics to a Prometheus-compatible server and logs to a Loki cluster. The Helm chart can also scrape additional metrics from kube-state-metrics, kubelet, and cAdvisor.

The Helm chart does not collect Prometheus node_exporter metrics; metrics from node_exporter must all have an instance label on them that has the same value as the instance label on Mimir metrics. For the list of necessary node_exporter metrics see the metrics prefixed with node in Grafana Cloud: Self-hosted Grafana Mimir integration.

You can configure your collection of metrics and logs by using the Grafana Agent operator. The Helm chart can install and use the Grafana Agent operator.

Note: Before the Helm chart can use the operator, you need to manually install all of the Kubernetes Custom Resource Definitions (CRDs) from the Grafana Agent operator YAML files.

It’s best to use the Grafana Agent operator for metrics and logs collection. However, if you prefer not to use it or you already have an existing Grafana Agent that you want to use, see Collect metrics and logs via Grafana Agent documentation in Grafana Mimir version 2.5.0.

Store credentials in a Secret:

If Prometheus and Loki are running without authentication, then you scan skip this section. Metamonitoring supports multiple ways of authentication for metrics and logs. If you are using a secret such as an API key to authenticate with Prometheus or Loki, then you need to create a Kubernetes Secret with that secret.

This is an example Kubernetes Secret:
YAML
```
apiVersion: v1
kind: Secret
metadata:
  name: metamonitoring-credentials
data:
  prometheus-api-key: FAKEACCESSKEY
  loki-api-key: FAKESECRETKEY
```
For information about how to create a Kubernetes Secret, see Creating a Secret.

Configure Helm chart values:

Merge the following YAML configuration into your Helm values file, and replace the values for url, username, passwordSecretName , and passwordSecretKey with the details of the Prometheus and Loki clusters, and the Secret that you created. If your Prometheus and Loki servers are running without authentication, then remove the auth blocks from the YAML below.

If you already have the Agent operator installed in your Kubernetes cluster, then set installOperator: false.

metaMonitoring:
  serviceMonitor:
    enabled: true
  grafanaAgent:
    enabled: true
    installOperator: true

    logs:
      remote:
        url: "https://example.com/loki/api/v1/push"
        auth:
          username: "12345"
          passwordSecretName: "metamonitoring-credentials"
          passwordSecretKey: "loki-api-key"

    metrics:
      remote:
        url: "https://example.com/api/v1/push"
        auth:
          username: "54321"
          passwordSecretName: "metamonitoring-credentials"
          passwordSecretKey: "prometheus-api-key"

      scrapeK8s:
        enabled: true
        kubeStateMetrics:
          namespace: kube-system
          labelSelectors:
            app.kubernetes.io/name: kube-state-metrics

For more information about using using the Helm chart to configure Grafana Mimir, refer to Run Grafana Mimir in production using the Helm chart.

Send metrics back into Mimir or GEM

You can also send the metrics that you collected (metamonitoring metrics) back into Mimir or GEM itself rather than sending them elsewhere.

When you leave the metamonitoring.grafanaAgent.metrics.remote.url field empty, then the chart automatically fills in the address of the GEM gateway Service or the Mimir NGINX Service.

If you have deployed Mimir, and metamonitoring.grafanaAgent.metrics.remote.url is not set, then the metamonitoring metrics are be sent to the Mimir cluster. You can query these metrics using the HTTP header X-Scope-OrgID: metamonitoring

If you have deployed GEM, then there are two alternatives:

If are using the trust authentication type (mimir.structuredConfig.auth.type=trust), then the same instructions apply as for Mimir.
If you are using the enterprise authentication type (mimir.structuredConfig.auth.type=enterprise, which is also the default when enterprise.enabled=true), then you also need to provide a Secret with the authentication token for the tenant.The token should be to an access policy with metrics:write scope. Assuming you are using the GEM authentication model, the Helm chart values should look like the following example.

metaMonitoring:
  serviceMonitor:
    enabled: true
  grafanaAgent:
    enabled: true
    installOperator: true

    metrics:
      remote:
        auth:
          username: metamonitoring
          passwordSecretName: gem-tokens
          passwordSecretKey: metamonitoring

Monitor without the Helm chart

To monitor the health of your system without using the Helm chart, see Collect metrics and logs without the Helm chart.

You can also use the self-hosted Grafana Cloud integration to monitor your Mimir system. Refer to Grafana Cloud: Self-hosted Grafana Mimir integration for more information.

About Grafana Mimir dashboards and alerts requirements

Tue, 22 Apr 2025 16:36:44 +0000

About Grafana Mimir dashboards and alerts requirements

Grafana Mimir dashboards and alerts require certain labels to exist on metrics scraped from Grafana Mimir.

The mimir-distributed Helm chart provides metamonitoring support, which takes care of these labels. For more information about Helm chart metamonitoring, refer to Collect metrics and logs via the Helm chart. If you are using Helm chart metamonitoring, go to Installing Grafana Mimir dashboards and alerts.

If you’re not using the Helm chart, you must configure your Prometheus or Grafana Alloy instance to add these labels for the dashboards and alerts to function. The following table shows the required label names and whether they can be customized when compiling dashboards or alerts from sources.

Label name	Configurable?	Description
`cluster`	Yes	The Kubernetes cluster or datacenter where the Mimir cluster is running. You can configure the cluster label via the `per_cluster_label` field in the mixin configuration.
`namespace`	Yes	The Kubernetes namespace where the Mimir cluster is running. You can configure the namespace label via the `per_namespace_label` field in the mixin configuration.
`job`	Yes	A prefix (by default `<namespace>/`) followed by the Mimir component. When running in monolithic mode, make sure that the `<component>` is `mimir`. When running in microservices mode, make sure that the `<component>` is the name of the specific Mimir component (singular), such as `distributor`, `ingester`, or `store-gateway`. Similarly, in read-write mode, make sure the `<component>` is either `mimir-read`, `mimir-write`, or `mimir-backend`. In the mixin configuration, you can configure the prefix via the `job_prefix` field, the label name via the `per_job_label` field, and the regular expressions that are used to match components via the `job_names` field.
`pod`	Yes	The unique identifier of a Mimir replica, for example the Pod ID when running on Kubernetes. You can configure the instance label via the `per_instance_label` field in the mixin configuration.
`instance`	Yes	The unique identifier of the node or machine where the Mimir replica is running, for example the node when running on Kubernetes. You can configure the node label via the `per_node_label` field in the mixin configuration.

For rules and alerts to function, you must configure your Prometheus or Grafana Alloy instance to scrape metrics from Grafana Mimir at an interval of 15s or shorter.

Deployment type

By default, Grafana Mimir dashboards assume Mimir is deployed in containers orchestrated by Kubernetes. If you’re running Mimir on baremetal, set the configuration field deployment_type: 'baremetal' and re-compile the dashboards.

Job selection

A metric could be exposed by multiple Grafana Mimir components, or even different applications running in the same namespace. To provide accurate dashboards and alerts, the job label (by default job) selects a metric from specific components. A job is a combination of a prefix and component. The default prefix is the Kubernetes namespace followed by a slash, for example <namespace>/ in <namespace>/ingester.

Pre-compiled dashboards and alerts are shipped with a default configuration. If you compile dashboards and alerts from source, you have the option to customize a) the label used for the job selection via the per_job_label field, b) the prefix expected in the label value, which you can omit by setting it to '' via the job_prefix field, and c) the regular expression used to select each Mimir component via the job_names field in the mixin configuration.

Default `job` selection in monolithic mode

When running Grafana Mimir in monolithic mode and using the pre-compiled dashboards and alerts, the job label should be set to <namespace>/mimir.

Default `job` selection in microservices mode

When running Grafana Mimir in microservices mode and using the pre-compiled dashboards and alerts, the job label should be set according to the following table.

Mimir service	Expected `job` label
Distributor	`<namespace>/distributor`
Ingester	`<namespace>/ingester`
Querier	`<namespace>/querier`
Ruler	`<namespace>/ruler`
Query-frontend	`<namespace>/query-frontend`
Query-scheduler	`<namespace>/query-scheduler`
Store-gateway	`<namespace>/store-gateway`
Compactor	`<namespace>/compactor`

Default `job` selection in read-write mode

When running Grafana Mimir in read-write mode and using the pre-compiled dashboards and alerts, set the job label accordingly:

Mimir service	Expected `job` label
Mimir Read	`<namespace>/mimir-read`
Mimir Write	`<namespace>/mimir-write`
Mimir Backend	`<namespace>/mimir-backend`

Additional resources metrics

The Grafana Mimir dashboards displaying CPU, memory, disk, and network resources utilization require Prometheus metrics scraped from the following endpoints:

For more information about the kubelet metrics and cAdvisor metrics exported by the kubelet, refer to Metrics For Kubernetes System Components.

Metrics from kubelet, kube-state-metrics, and cAdvisor must all have a cluster label with the same value as in the Mimir metrics.

Metrics from Node Exporter and cAdvisor must all have an instance label on them that has the same value as the instance label on Mimir metrics.

Log labels

The Slow queries dashboard uses a Loki data source with the logs from Grafana Mimir to visualize slow queries. The query-frontend component logs query statistics when the -query-frontend.query-stats-enabled parameter is set to true. These logs need to have specific labels in order for the dashboard to work.

Label name	Configurable?	Description
`cluster`	Yes	The Kubernetes cluster or datacenter where the Mimir cluster is running. You can configure the cluster label via the `per_cluster_label` field in the mixin configuration.
`namespace`	No	The Kubernetes namespace where the Mimir cluster is running.
`name`	Yes	Name of the component. For example, `query-frontend`. You can configure the cluster label via the `per_component_loki_label` field in the mixin configuration.

Deploying the Grafana Mimir monitoring mixin

Tue, 22 Apr 2025 16:36:44 +0000

Deploying the Grafana Mimir monitoring mixin

Grafana Mimir exposes a /metrics endpoint returning Prometheus metrics. You can configure your Prometheus server to scrape Grafana Mimir or you can use the built-in functionality of the Helm chart to automatically send these metrics to a remote. The endpoint is exposed on the Mimir HTTP server address / port which can be customized through -server.http-listen-address and -server.http-listen-port CLI flags or their respective YAML config options.

Dashboards and alerts

Grafana Mimir is shipped with a comprehensive set of production-ready Grafana dashboards and alerts to monitor the state and health of a Mimir cluster.

Dashboards provide both a high-level and in-depth view of every aspect of a Grafana Mimir cluster. You can take a look at all the available dashboards in this overview.

Alerts allow you to monitor the health of a Mimir cluster. For each alert, we provide detailed runbooks to further investigate and fix the issue.

The requirements documentation lists prerequisites for using the Grafana Mimir dashboards and alerts.

The installation instructions show available options to install Grafana Mimir dashboards and alerts.

Installing Grafana Mimir dashboards and alerts

Tue, 22 Apr 2025 16:36:44 +0000

Installing Grafana Mimir dashboards and alerts

Grafana Mimir is shipped with a comprehensive set of production-ready Grafana dashboards and alerts to monitor the state and health of a Mimir cluster.

Requirements

Grafana Mimir dashboards and alerts require specific labels to be set by Prometheus or Grafana Alloy when scraping your Mimir cluster metrics
Some dashboards require recording rules that you should install in your Prometheus

Install from package

Grafana Mimir provides ready to use Grafana dashboards in the .json format and Prometheus alerts in the .yaml format, that you can directly import into your Grafana installation and Prometheus config.

The packaged dashboards and alerts have been compiled from the sources using a default configuration and don’t allow you to customize the required metrics label names. If you need to customize the required metrics label names please choose one of the other installation options.

Download dashboards, recording rules and alerts from Grafana Mimir repository
Import dashboards in Grafana
Install recording rules and alerts in your Prometheus

Install from sources

Grafana Mimir dashboards and alerts are built using Jsonnet language and you can compile them from sources. If you choose this option, you can change the configuration to match your deployment, like customizing the required label names.

Checkout Mimir source code

git clone https://github.com/grafana/mimir.git

Review the mixin configuration at operations/mimir-mixin/config.libsonnet, and apply your changes if necessary.
Compile the mixin
Bash
```
make build-mixin
```
Import the dashboards saved at operations/mimir-mixin-compiled/dashboards/ in Grafana
Install the recording rules saved at operations/mimir-mixin-compiled/rules.yaml in your Prometheus
Install the alerts saved at operations/mimir-mixin-compiled/alerts.yaml in your Prometheus

Install dashboards from Jsonnet mixin

In case you’re already using Jsonnet to define your infrastructure as a code, you can vendor the Grafana Mimir mixin directly into your infrastructure repository and configure it overriding the _config fields. Given the exact setup really depends on a case-by-case basis, the following instructions are not meant to be prescriptive but just show the main steps required to vendor the mixin.

Initialize Jsonnet
Bash
```
jb init
```

Install Grafana Mimir mixin

jb install github.com/grafana/mimir/operations/mimir-mixin@main

Import and configure it

(import 'github.com/grafana/mimir/operations/mimir-mixin/mixin.libsonnet') + {
  _config+:: {
    // Override the Grafana Mimir mixin config here.
  },
}

Deploy mixin with Terraform

Based on Jsonnet configuration file, you can use Terraform to deploy the mixin in both Grafana and Grafana Mimir. To do so, you can use grafana/grafana, ovh/mixtool, and ovh/mimirtool providers.

Create a Terraform main.tf file:

# Specify which providers to use
terraform {
  required_version = ">= 1.3.5"
  required_providers {
    mixtool = {
      source  = "ovh/mixtool"
      version = "~> 0.1.1"
    }
    mimirtool = {
      source  = "ovh/mimirtool"
      version = "~> 0.1.1"
    }
    grafana = {
      source  = "grafana/grafana"
      version = "~> 1.32.0"
    }
  }
}

# Configure providers if needed
provider "grafana" {
  url  = "http://localhost:9000"
  auth = "admin:admin"
}

provider "mimirtool" {
  address   = "http://localhost:9009"
  tenant_id = "anonymous"
}

locals {
  mixin_source   = "custom.libsonnet"
  jsonnet_path   = "vendor"
}

# Build alerts
data "mixtool_alerts" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Build rules
data "mixtool_rules" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Build dashboards
data "mixtool_dashboards" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Deploy rules
resource "mimirtool_ruler_namespace" "rules" {
  namespace   = "rules_community"
  config_yaml = data.mixtool_rules.mimir.rules
}

# Deploy alerts
resource "mimirtool_ruler_namespace" "alerts" {
  namespace   = "alerts_community"
  config_yaml = data.mixtool_alerts.mimir.alerts
}

# Deploy dashboards
resource "grafana_dashboard" "mimir" {
  for_each    = data.mixtool_dashboards.mimir.dashboards
  config_json = each.value
}

Initialize Terraform:
Bash
```
terraform init
```
Review the changes that Terraform would apply to your infrastructure:
Bash
```
terraform plan
```
Deploy the changes:
Bash
```
terraform apply
```

Viewing Grafana Mimir dashboards

Tue, 22 Apr 2025 16:36:44 +0000

Viewing Grafana Mimir dashboards

Grafana Mimir provides the following production-ready dashboards.

Collecting metrics and logs from Grafana Mimir

Tue, 22 Apr 2025 17:05:43 +0000

Collecting metrics and logs from Grafana Mimir

You can collect logs and metrics from a Mimir or GEM cluster. To set up dashboards and alerts, see Installing Grafana Mimir dashboards and alerts or Grafana Cloud: Self-hosted Grafana Mimir integration.

Install Grafana Mimir using the Helm chart

As a best practice, use the Grafana Mimir Helm chart to install a Mimir or GEM cluster. When you deploy the Helm chart, ensure that you’ve enabled ServiceMonitor objects. For more information, refer to the documentation for the Grafana Mimir Helm chart.

After installing the cluster, deploy the Grafana Kubernetes Monitoring Helm chart, which contains instances of Grafana Alloy for collecting metrics and logs. These Alloy instances detect the ServiceMonitor objects deployed as part of the Mimir Helm chart. For more information, refer to Grafana Kubernetes Monitoring Helm chart.

Note
The Grafana Mimir Helm chart contains built-in configurations for Grafana Agent, which is now deprecated. Use the Grafana Alloy configurations included in the Kubernetes Monitoring Helm chart instead.

Install Grafana Mimir without the Helm chart

You can still use the dashboards and rules in the monitoring-mixin, even if you’re not deploying Mimir or GEM via the Helm chart. If you’re not using the Helm chart, start by using the Grafana Alloy configuration from Collect metrics and logs via Grafana Alloy. It’s possible that you need to modify this configuration. For more information, see dashboards and alerts requirements.

Service discovery

As a best practice, deploy Grafana Mimir on Kubernetes. The Grafana Alloy configuration relies on Kubernetes service discovery and Pod labels to constrain the collected metrics and logs to ones that are strictly related to the Grafana Mimir deployment. If you are deploying Grafana Mimir on something other than Kubernetes, then replace the discovery.kubernetes component with another Alloy component that can discover the Mimir processes.

Collect metrics and logs via Grafana Alloy

Set up Grafana Alloy to collect logs and metrics from Mimir or GEM. To get started with Grafana Alloy, refer to Get started with Grafana Alloy. After deploying Alloy, refer to Collect and forward Prometheus metrics for instructions on how to configure your Alloy instance to scrape Mimir or GEM.

Monitor Grafana Mimir on Grafana Labs

Monitor the health of your system

Monitor the health of your system

Configure the Grafana Agent operator via the Helm chart

Send metrics back into Mimir or GEM

Monitor without the Helm chart

About Grafana Mimir dashboards and alerts requirements

About Grafana Mimir dashboards and alerts requirements

Deployment type

Job selection

Default job selection in monolithic mode

Default job selection in microservices mode

Default job selection in read-write mode

Additional resources metrics

Log labels

Deploying the Grafana Mimir monitoring mixin

Deploying the Grafana Mimir monitoring mixin

Dashboards and alerts

Installing Grafana Mimir dashboards and alerts

Installing Grafana Mimir dashboards and alerts

Requirements

Install from package

Install from sources

Install dashboards from Jsonnet mixin

Deploy mixin with Terraform

Viewing Grafana Mimir dashboards

Viewing Grafana Mimir dashboards

Collecting metrics and logs from Grafana Mimir

Collecting metrics and logs from Grafana Mimir

Install Grafana Mimir using the Helm chart

Install Grafana Mimir without the Helm chart

Service discovery

Collect metrics and logs via Grafana Alloy

Default `job` selection in monolithic mode

Default `job` selection in microservices mode

Default `job` selection in read-write mode