Remote configuration on Grafana Labs

The scaling challenge

Tue, 18 Aug 2026 05:43:38 -0700

The scaling challenge

Say you need to add a new metrics scrape to your collectors, maybe you’re rolling out a new integration or collecting extra data to troubleshoot an incident. With the manual approach, you access each machine, edit its configuration file, and reload the collector, then repeat for the next one. For 5 collectors that’s tedious. For 50 it’s most of a day. For 500 it’s impossible to do consistently.

Configuration management tools like Ansible or Terraform help, but they don’t remove the work. You’re still managing deployments, testing changes, and coordinating rollouts, so every configuration change becomes its own project.

How remote configuration changes this

Remote configuration flips the model. You create a configuration pipeline once in Fleet Management and assign it to collectors by their attributes, such as all production collectors or everything in us-east. The collectors pull the new configuration automatically on their next poll cycle.

Updates per change

Approach	5 collectors	50 collectors	500 collectors
Manual	5 updates	50 updates	500 updates
Remote config	1 update	1 update	1 update

Configure once, deploy everywhere. One update reaches your entire fleet.

Configuration pipelines

Tue, 18 Aug 2026 05:43:38 -0700

Configuration pipelines

A configuration pipeline is the unit you create in Fleet Management. When you configure a collector remotely, you’re creating at least one pipeline. Each pipeline holds standard collector configuration for a single operation, such as scraping MySQL metrics, built from components like prometheus.scrape, loki.source.file, or prometheus.remote_write.

You match a pipeline to collectors by their attributes. One pipeline can match many collectors, and one collector can match many pipelines. Each collector pulls the set of pipelines that match it, and that set is its remote configuration.

Each pipeline has a name, configuration content, and matching attributes.

Pipeline components

Component	Description
Name	Unique identifier for the pipeline
Content	Standard collector components for one operation
Matching attributes	Rules that assign the pipeline to collectors

Pipelines are reusable. Create once, assign to many collectors using attribute matching.

Working with pipelines

Tue, 18 Aug 2026 05:43:38 -0700

Creating pipelines

There are two ways to create a pipeline. Start from a Grafana Cloud integration to get going quickly, or write your own configuration from scratch when you have specific requirements.

Approach	Best for
Grafana Cloud integration	Common use cases (Linux, MySQL, PostgreSQL)
Custom configuration	Specific requirements, custom scrapes

Pipeline operations

Once a pipeline exists, several operations are available. Deactivate a pipeline to temporarily stop collecting certain metrics, edit its content at any time, or view its history, which is invaluable for troubleshooting.

Action	What it does
Activate/Deactivate	Toggle collection on/off instantly
Edit	Update configuration content
View history	See what changed and when

Why this matters

Create a pipeline once, match it to collectors with attributes, and Fleet Management distributes it. One pipeline can serve hundreds of collectors.

Instant changes: Changes apply with no deployments or restarts.
Safe experimentation: Toggle off if something goes wrong.
Audit trail: History shows what changed and when.

Designing your attribute schema

Tue, 18 Aug 2026 05:43:38 -0700

How to choose your attributes

A little planning up front saves a lot of pain later. Start by asking one question: “What groups of collectors should receive the same configuration?”

To match a pipeline to…	You need an attribute like…
All production servers	`env=production`
Collectors in specific regions	`region=us-east`
Collectors owned by a team	`team=platform`
Collectors for a specific service	`service=api-gateway`

Common attribute patterns

Attribute	Example values	Use case
`env`	`production`, `staging`, `dev`	Environment-specific configs
`region`	`us-east`, `eu-west`, `ap-south`	Region-specific collection
`team`	`platform`, `backend`, `data`	Team ownership
`service`	`api`, `web`, `worker`	Service-specific monitoring

Start simple. Begin with env and one or two others. Add more as patterns emerge from real usage.

Attribute mistakes to avoid

Tue, 18 Aug 2026 05:43:38 -0700

What not to do

Attributes should describe groups of collectors, not individual instances. Most mistakes come from ignoring that.

Mistake	Why it’s a problem
Too many attributes	Hard to manage, and hits the limits (22 local per API request, 100 remote per collector)
Frequently changing values	Use remote attributes in UI instead
Sensitive information	Attributes are visible in the Fleet Management UI
Duplicating reserved attributes	`collector.os`, `collector.version`, and `collector.ID` already exist
Inconsistent naming	`myEnv` vs `my_env` vs `my-env` causes confusion
Overly specific values	`hostname=server-123` (use `collector.ID` instead)

Tips for beginners

Tip	Why it helps
Start simple	Begin with `env` and maybe one or two others
Be consistent	Pick a naming convention and stick to it
Document your schema	Note down what each attribute means
Think about matching	Every attribute should help you match pipelines to collectors

Attribute examples

Tue, 18 Aug 2026 05:43:38 -0700

Examples

Seeing real configurations makes attributes concrete. These two Alloy examples move from a minimal starting schema to a more complete one for a larger deployment.

Example 1: Minimal starting schema

This example shows the recommended starting point for new Fleet Management users: just two attributes:

remotecfg {
    // ... other configuration ...
    
    attributes = {
        "env"  = "production",   // Which environment: production, staging, or dev
        "team" = "platform"      // Who owns this collector
    }
}

Attribute	Value	What it means
`env`	`production`	This collector monitors production infrastructure.
`team`	`platform`	The platform team owns and manages this collector.

What you can match with this schema

All production collectors → env=production
All collectors owned by platform team → team=platform
Production collectors owned by platform → env=production, team=platform

Example 2: Expanded schema for larger deployments

As your fleet grows, you may need more precise matching. This example adds a region attribute:

remotecfg {
    // ... other configuration ...
    
    attributes = {
        "env"    = "production",
        "region" = "us-east-1",
        "team"   = "platform"
    }
}

Attribute	Value	What it means
`env`	`production`	This collector monitors production infrastructure.
`region`	`us-east-1`	This collector is located in the US East region.
`team`	`platform`	The platform team owns and manages this collector.

What you can match with this schema

All US East collectors → region=us-east-1
Production collectors in US East → env=production, region=us-east-1
Platform team’s US East production collectors → all three attributes

Both examples define local attributes in the local configuration, but you can also set attributes through the Fleet Management API or the user interface. Notice the naming pattern in each: keys are lowercase with no spaces, and values are short, descriptive strings.

Start simple, expand as needed. Begin with Example 1. Add more attributes when you need that level of matching precision.

Attribute matching

Tue, 18 Aug 2026 05:43:38 -0700

Attribute matching

Attribute matching is the mechanism that connects pipelines to collectors. When you create a pipeline, you define matchers, which are rules that specify which collectors should receive that pipeline based on their attributes.

Matchers determine which collectors receive a pipeline.

How matchers work

You define…	To match…
`env=production`	All collectors with `env` attribute set to `production`
`region=~"us-.*"`	All collectors in any US region

Matcher syntax

Each matcher combines an attribute name, an operator, and a value. Matchers follow the same syntax as Prometheus Alertmanager matchers, so they may look familiar.

Operator	Meaning	Example
`=`	Equals	`env=production`
`!=`	Not equals	`env!=development`
`=~`	Regex match	`region=~"us-.*"`
`!~`	Regex not match	`team!~"test-.*"`

Matching rule: AND logic

When you assign multiple matchers to a pipeline, Fleet Management combines them with AND logic. All matchers must match for a collector to receive the pipeline.

Matchers defined	Collector receives pipeline if…
`env=prod`	`env` is `prod`
`env=prod`, `region=us-east`	`env` is `prod` AND `region` is `us-east`

This lets you be precise about which collectors receive a configuration. The more matchers you add, the narrower your matching becomes.

Matching in practice

Tue, 18 Aug 2026 05:43:38 -0700

Common patterns

Because every collector carries its own attributes, you can organize your fleet along whatever dimensions fit: environment, region, team, service, or even Kubernetes namespace.

Pattern	Matcher examples
Environment-based	`env="production"`, `env="staging"`
Region-based	`region="us-east"`, `region="eu-west"`
Team-based	`team="platform"`, `team="backend"`
Catch-all	`collector.os=~".+"` (matches all collectors)

Example: Production platform pipeline

Matchers: env=~"prod-.*", team="platform"

This pipeline applies to collectors where:

env matches the regex prod-.* AND
team equals platform

Limits to know

Limit	Value
Matcher length	200 characters max (key + operator + value)
Local attributes per collector	22 max when polling

These limits are generous for most use cases, but worth knowing as you design your schema.

Configuration strategies

Tue, 18 Aug 2026 05:43:38 -0700

Configuration strategies

Fleet Management gives you flexibility in how you balance local and remote configuration. You don’t have to go all-in on either approach. Choose the strategy that fits your situation.

Remote-first

Minimal local config, everything else managed remotely. Best when a dedicated team manages observability and you want maximum central control over the fleet.

remotecfg {
    url = "https://..."
    // ... auth config
    id = constants.hostname
    attributes = {"env" = "prod"}
}

// Everything else comes from Fleet Management

Local-first

Primary config stays local, Fleet Management for self-monitoring. Best when your main configuration is already managed through an existing deployment process and you want Fleet Management mainly for collector health.

// Local config: full configuration
remotecfg {
    url = "https://..."
    id = constants.hostname
    attributes = {"env" = "prod"}
}

// All your standard collection is here
prometheus.scrape "node" { /* ... */ }
prometheus.scrape "app" { /* ... */ }
loki.source.file "logs" { /* ... */ }

// Fleet Management used only for collector self-monitoring

Hybrid

Top-level config blocks stay local. Everything else is remote. Best when some configuration must stay local but your needs vary by environment, so the rest is easier to manage remotely.

// Local config: top-level blocks
logging { level = "info" }

livedebugging { enabled = true }

remotecfg {
    url = "https://..."
    id = constants.hostname
    attributes = {"env" = "prod"}
}

// All other config managed remotely in Fleet Management

Why it's safe to experiment

Tue, 18 Aug 2026 05:43:38 -0700

Safeguards

Changing configuration across a fleet can feel risky. Nobody wants to push a bad config, stop data collection, or make an incident worse. Fleet Management is designed with safety in mind, with four safeguards that make it safe to experiment.

Four safeguards

Safeguard	What it means	Why it matters
Isolation	Local and remote Alloy configs run independently	Remote errors don’t break local collection.
Fallback	Invalid Alloy configs aren’t applied	Bad syntax doesn’t stop your collector.
Instant rollback	Deactivate pipelines with one click	Recovery from any problem is fast.
Staged rollout	Use matchers to limit how many collectors are affected	Test on one collector before fleet-wide.

What happens when things go wrong

Scenario	What Alloy does	Your action
Syntax error in pipeline	Rejects config, keeps last known good	Fix syntax, re-save pipeline.
Pipeline causes high load	Runs but collector health dashboards show warning	Deactivate pipeline, investigate.
Wrong collectors matched	Pipeline runs on unintended collectors	Update matchers, deactivate if needed.

Isolation and automatic fallback are Grafana Alloy behaviors. The OpenTelemetry Collector merges local and remote configuration and doesn’t automatically roll back, so validate changes carefully when using it.

Safe experimentation workflow

Tue, 18 Aug 2026 05:43:38 -0700

Safe experimentation workflow

Create pipeline
Set matcher for ONE test collector (e.g., collector.ID=“test-collector-01”)
Activate and verify (no errors, acceptable performance)
Expand matcher to small group (e.g., env=“staging”)
Verify at scale
Expand to full target (e.g., env=“production”)

Why this is safe

Isolation + Fallback + Instant rollback + Staged rollout = Safe to experiment

You’re not gambling with production. You’re making controlled changes with multiple safety nets. Start small, verify, expand. If something goes wrong at any step, deactivate and try again. Recovery is fast and doesn’t require deployments or restarts.

Coming up next

You’ll categorize your collector fleet in the path. You’ll assign attributes like team, service.name, and env, the foundation that makes precise configuration and safe rollouts possible.

Categorize your collector fleet

Tue, 18 Aug 2026 05:43:38 -0700

Categorize your collector fleet

Learning path

Categorize a collector fleet

Welcome to the Grafana Fleet Management learning path that shows you how to organize and categorize your collector fleet using custom attributes for remote configuration. Learn to categorize your infrastructure with meaningful attributes that answer critical questions such as who owns this collector, which service does it support, and where is it running.

9 min

Intermediate

Docs & blog posts

Open in Grafana Cloud

Complete this learning path directly in your Grafana Cloud stack with an interactive learning experience.

Open in Grafana Cloud

View on website

Key takeaways

Tue, 18 Aug 2026 05:43:38 -0700

Key takeaways

Fleet Management = Central control for all your collectors.
Collectors register with remotecfg block and appear in inventory.
Configuration pipelines remotely configure without touching machines.
Attribute matching matches pipelines to specific collector groups.
Local + remote configs run in parallel, isolated from each other, which makes it safe to experiment.
Scales your operations so managing many collectors takes the same effort as a few.

Remote configuration on Grafana Labs

The scaling challenge

The scaling challenge

How remote configuration changes this

Updates per change

Configuration pipelines

Configuration pipelines

Pipeline components

Working with pipelines

Creating pipelines

Pipeline operations

Why this matters

Designing your attribute schema

How to choose your attributes

Common attribute patterns

Attribute mistakes to avoid

What not to do

Tips for beginners

Attribute examples

Examples

Example 1: Minimal starting schema

What you can match with this schema

Example 2: Expanded schema for larger deployments

What you can match with this schema

Attribute matching

Attribute matching

How matchers work

Matcher syntax

Matching rule: AND logic

Matching in practice

Common patterns

Example: Production platform pipeline

Limits to know

Configuration strategies

Configuration strategies

Remote-first

Local-first

Hybrid

Why it's safe to experiment

Safeguards

Four safeguards

What happens when things go wrong

Safe experimentation workflow

Safe experimentation workflow

Why this is safe

Coming up next

Categorize your collector fleet

Categorize your collector fleet

Categorize a collector fleet

Open in Grafana Cloud

Key takeaways

Key takeaways

The core workflow

Resources

Documentation

Community