Configure Grafana Mimir Helm chart on Grafana Labs

Configure Grafana Mimir autoscaling with Helm

Fri, 22 Aug 2025 09:49:49 +0000

Configure Grafana Mimir autoscaling with Helm

Warning
Autoscaling support in the Helm chart is currently experimental. Use with caution in production environments and thoroughly test in a non-production environment first.

You can configure autoscaling for Mimir components using (Kubernetes Event-driven Autoscaling) KEDA and Kubernetes Horizontal Pod Autoscaler (HPA).

Before you begin

Ensure you have a running Mimir cluster deployed with Helm.
Verify you have the required permissions to modify Helm deployments.
Familiarize yourself with Kubernetes HPA concepts.

Prerequisites

To use autoscaling, you need:

KEDA installed in your Kubernetes cluster
Prometheus metrics available for scaling decisions

Warning
Don’t use the same Mimir or Grafana Enterprise Metrics cluster for storing and querying autoscaling metrics. Using the same cluster can create a dangerous feedback loop.

For instance, if the Mimir or GEM cluster becomes unavailable, autoscaling stops working, because it cannot query the metrics. This prevents the cluster from automatically scaling up during high load or recovery. This inability to scale further exacerbates the cluster’s unavailability, which might, in turn, prevent the cluster from recovering.

Instead, use a separate Prometheus instance or a different metrics backend for autoscaling metrics.

Supported components

The Mimir Helm chart supports autoscaling for the following components:

About KEDA

KEDA is a Kubernetes operator that simplifies the setup of HPA with custom metrics from Prometheus. It consists of:

An operator and external metrics server
Support for multiple metric sources, including Prometheus
Custom resources (ScaledObject) that define scaling parameters
Automatic HPA resource management

For more information, refer to the KEDA documentation.

Configure autoscaling for a new installation

Follow these steps to enable autoscaling when deploying Mimir for the first time.

Steps

Configure the Prometheus metrics source in your values file:

kedaAutoscaling:
  prometheusAddress: "http://prometheus.monitoring:9090"
  pollingInterval: 10

Enable and configure autoscaling for desired components:

querier:
  kedaAutoscaling:
    enabled: true
    minReplicaCount: 2
    maxReplicaCount: 10

Deploy Mimir using Helm:

helm upgrade --install mimir grafana/mimir-distributed -f values.yaml

Expected outcome

After deployment:

KEDA creates ScaledObject resources for configured components.
HPA resources are automatically created and begin monitoring metrics.
Components scale based on configured thresholds and behaviors.

Migrate existing deployments to autoscaling

Follow these steps to enable autoscaling for an existing Mimir deployment.

Warning
Autoscaling support in the Helm chart is currently experimental. Migrating to autoscaling carries risks for cluster availability.

Enabling autoscaling removes the replicas field from deployments. If KEDA/HPA hasn’t started autoscaling a deployment yet, Kubernetes interprets no replicas as meaning 1 replica. This can cause an outage if the transition is not handled carefully. If you’re using GitOps tools like FluxCD or ArgoCD, you might need to take additional steps to manage the transition.

Consider testing the migration in a non-production environment first.

Before you begin

Back up your current Helm values.
Plan for potential service disruption.
Consider testing in a non-production environment first.
Ensure you have a rollback plan ready.
Consider migrating one component at a time to minimize risk.

Steps

Add the autoscaling configuration with preserveReplicas enabled:

querier:
  kedaAutoscaling:
    enabled: true
    preserveReplicas: true # Maintains stability during migration
    # ... autoscaling configuration ...

Apply the changes and verify the KEDA setup:

# Apply changes
helm upgrade mimir grafana/mimir-distributed -f values.yaml

# Verify setup
kubectl get hpa
kubectl get scaledobject
kubectl describe hpa

Wait 2-3 polling intervals to confirm that KEDA is managing scaling.

Remove preserveReplicas.

querier:
  kedaAutoscaling:
    enabled: true
    # Remove preserveReplicas

Apply the updated configuration:

helm upgrade mimir grafana/mimir-distributed -f values.yaml

Troubleshooting

If pods scale down to 1 replica after removing preserveReplicas:

Revert changes:

querier:
  kedaAutoscaling:
    enabled: true
    preserveReplicas: true

Verify KEDA setup:
- Check HPA status
- Verify metrics are being received
- Check for conflicts with other tools
- Ensure enough time was given for KEDA to take control (at least 2-3 polling intervals)
Try migrating again after resolving issues.

Note
If you’re using GitOps tools like FluxCD or ArgoCD, they might try to reconcile the state and conflict with HPA’s scaling decisions. Consult your GitOps tool’s documentation for handling HPA transitions.

Monitor autoscaling health

The following conditions indicate unhealthy autoscaling:

KEDA operator is down: ScaledObject changes don’t propagate to HPA.
KEDA metrics server is down: HPA can’t receive updated metrics.
HPA is unable to scale: MimirAutoscalerNotActive alert fires.

For production deployments, configure high availability for KEDA.

For more information about monitoring autoscaling, refer to Monitor Grafana Mimir.

Configure GEM federation-frontend

Fri, 22 Aug 2025 09:49:49 +0000

Configure GEM federation-frontend

You can deploy the Grafana Enterprise Metrics (GEM) federation-frontend in a Kubernetes cluster using Helm. The federation-frontend allows you to query data from multiple GEM clusters using a single endpoint. For more information about cluster query federation, refer to the federation-frontend documentation.

Note
This guide focuses specifically on deploying the federation-frontend component as a standalone deployment without any additional GEM or Mimir components.

Before you begin

Set up a GEM cluster: For information about setting up and configuring a GEM deployment, refer to Deploy Grafana Enterprise Metrics using the Helm chart.
Provision an access token with the metrics:read scope for each cluster that you want to query. For more information, refer to Set up a GEM tenant.

Deploy the GEM federation-frontend

Create a Kubernetes Secret named gem-tokens with the GEM access tokens for each of the remote GEM clusters. The Helm values file uses this Secret later. Replace TOKEN1 and TOKEN2 with the access tokens for the remote GEM clusters.
YAML
```
apiVersion: v1
kind: Secret
metadata:
  name: gem-tokens
data:
  CLUSTER_1_GEM_TOKEN: TOKEN1
  CLUSTER_2_GEM_TOKEN: TOKEN2
```
Apply the secret to your cluster in the federation-frontend-demo namespace with the kubectl command:
Bash
```
kubectl -n federation-frontend-demo apply -f mysecret.yaml
```

Create a values file named federation-frontend.yaml with the following content.

Replace http://gem-query-frontend.monitoring.svc.cluster.local:8080/prometheus and https://gem.monitoring.acme.local/prometheus with the URLs of the GEM clusters you want to query. Replace tenant-1 and tenant-2 with the tenant IDs of the remote GEM clusters.

Note that these resource settings are examples that are sufficient for small deployments. Adjust the values based on your specific requirements and load:

# Enable enterprise features
enterprise:
  enabled: true

# Enable and configure federation-frontend
federation_frontend:
  enabled: true
  # Since this is a standalone deployment, configure the chart to not render any of the other GEM components.
  disableOtherComponents: true
  replicas: 2
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 1
      memory: 256Mi
  extraEnvFrom:
    - secretRef:
        name: gem-tokens

# Configure the remote GEM clusters to query.
mimir:
  structuredConfig:
    auth:
      type: trust
    # The federation-frontend doesn't handle authentication or authorization. Disabling multitenancy means the federation-frontend doesn't require the X-Scope-OrgID header.
    # With disabled multitenancy, the federation-frontend uses the authn/z material from the proxy_targets configuration.
    # For a complete list of configuration options, refer to the configuration reference at https://grafana.com/docs/enterprise-metrics/<GEM_VERSION>/config/reference/#federation.
    multitenancy_enabled: false
    federation:
      proxy_targets:
        - name: "cluster-1"
          url: "http://gem-query-frontend.monitoring.svc.cluster.local:8080/prometheus"
          basic_auth:
            username: tenant-1
            password: "${CLUSTER_1_GEM_TOKEN}"

        - name: "cluster-2"
          url: "https://gem.monitoring.acme.local/prometheus"
          basic_auth:
            username: tenant-2
            password: "${CLUSTER_2_GEM_TOKEN}"

# Disable MinIO
minio:
  enabled: false

# The federation-frontend doesn't need the rollout-operator for rollouts, so it can be disabled.
rollout_operator:
  enabled: false

Deploy the federation-frontend using helm:
Bash
```
helm install federation-frontend grafana/mimir-distributed -f federation-frontend.yaml -n federation-frontend-demo
```
This deploys only the federation-frontend component. The federation-frontend is configured to proxy queries to the GEM clusters that you specify in the proxy_targets configuration.
Verify that the federation-frontend is running. The simplest way to do this is to issue a label names query against the federation-frontend service.

This example tries to reach the Kubernetes service from the cluster and request the label names from the past 12 hours.
Bash
```
curl -XPOST 'https://mimir-federation-frontend:8080/prometheus/api/v1/labels' \
  -d "start=$(date -u +%Y-%m-%dT%H:%M:%S.0Z -d '12 hours ago' 2>/dev/null || date -u -v -12H +%Y-%m-%dT%H:%M:%S.0Z)" \
  -d "end=$(date -u +%Y-%m-%dT%H:%M:%S.0Z)"
```
You should receive a response with the label names from the remote GEM clusters similar to this:
JSON
```
{
  "status": "success",
  "data": ["__cluster__", "__name__", "hash_extra", "series_id"]
}
```

Configure Grafana Mimir to allow Vault Agent to inject certificates and keys into Pods

Fri, 22 Aug 2025 09:49:49 +0000

Configure Grafana Mimir to allow Vault Agent to inject certificates and keys into Pods

When you enable this feature, Helm updates the annotations on all Pods that have TLS (transport layer security) configurable components. When you deploy Mimir or GEM, Vault Agent fetches the relevant secrets from Vault according to the annotations and mounts them to the Pod.

Note: Vault and Vault Agent must to be running already.

Example values.yaml file:

vaultAgent:
  enabled: true
  roleName: "test-role"
  clientCertPath: "client/cert/path"
  clientKeyPath: "client/key/path"
  serverCertPath: "server/cert/path"
  serverKeyPath: "server/key/path"
  caCertPath: "ca/cert/path"

Generated Kubernetes Deployment YAML file based on the preceding example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: release-name
  labels:
    helm.sh/chart: mimir-release-1.2.3
    app.kubernetes.io/name: mimir
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/component: comonent-name
    app.kubernetes.io/version: "123"
    app.kubernetes.io/managed-by: Helm
  annotations: {}
  namespace: "default"
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: mimir
      app.kubernetes.io/instance: release-name
      app.kubernetes.io/component: component-name
  strategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "test-role"
        vault.hashicorp.com/agent-inject-secret-client.crt: "client/cert/path"
        vault.hashicorp.com/agent-inject-secret-client.key: "client/key/path"
        vault.hashicorp.com/agent-inject-secret-server.crt: "server/cert/path"
        vault.hashicorp.com/agent-inject-secret-server.key: "server/key/path"
        vault.hashicorp.com/agent-inject-secret-root.crt: "ca/cert/path"

vault.hashicorp.com/agent-inject-secret-<FILENAME>: '<PATH>' tells Vault Agent where to find the secret, and the name of the file to write the secret to. For example: vault.hashicorp.com/agent-inject-secret-client.crt: 'client/cert/path' will look for the secret at the path client/cert/path within Vault, and mount this secret to the pod as client.crt in the /vault/secrets/ directory.

For more information about Vault and Vault Agent, see Injecting Vault Secrets Into Kubernetes Pods via a Sidecar.

To configure TLS in Mimir, refer to Securing Grafana Mimir communications with TLS.

Configure native histograms

Fri, 22 Aug 2025 09:49:49 +0000

Configure native histograms

To enable support for ingesting Prometheus native histograms over the remote write API endpoint, set the configuration parameter native_histograms_ingestion_enabled to true.

To enable support for querying native histograms together with Grafana Mimir query sharding, set the configuration parameter query_result_response_format to protobuf.

Example values file:

mimir:
  structuredConfig:
    frontend:
      query_result_response_format: protobuf
    limits:
      native_histograms_ingestion_enabled: true

Note
Native histograms are an experimental feature of Grafana Mimir.

To configure bucket limits for native histograms, refer to Configure native histograms.

To configure Grafana Agent or Prometheus to write native histograms to Grafana Mimir, refer to Send native histograms to Mimir.

To visualize native histograms in Mimir, refer to Visualize native histograms.

Configure Redis cache

Fri, 22 Aug 2025 09:49:49 +0000

Configure Redis cache

Caution
Starting with Mimir version 2.14, the experimental support for Redis caching is deprecated. It will be removed in a future release. You are encouraged to switch to Memcached.

Besides support for Memcached, Mimir also supports Redis for the chunks-cache, index-cache, results-cache and metadata-cache. To use Redis, deploy Redis instances, disable the built-in Memcached configuration flag in values.yaml of mimir-distributed Helm chart, and then configure Mimir to use Redis.

To disable Memcached, remove any chunks-cache, index-cache, metadata-cache and results-cache configuration from your Helm values.yaml file. Alternatively, explicitly disable each of the Memcached instances by setting enabled to false:

chunks-cache:
  enabled: false
index-cache:
  enabled: false
metadata-cache:
  enabled: false
results-cache:
  enabled: false

Next, configure Mimir to connect to Redis using structuredConfig. Refer to the configuration parameters reference for Redis connection configuration options. For example:

mimir:
  structuredConfig:
    blocks_storage:
      bucket_store:
        chunks_cache:
          backend: redis
          redis:
            endpoint: <redis-url>:6379
        index_cache:
          backend: redis
          redis:
            endpoint: <redis-url>:6379
        metadata_cache:
          backend: redis
          redis:
            endpoint: <redis-url>:6379
    frontend:
      cache_results: true
      results_cache:
        backend: redis
        redis:
          endpoint: <redis-url>:6379