--- title: "Set up private probes | Grafana Cloud documentation" description: "How to configure your internal synthetic monitoring agents (private probes) in Grafana Synthetic Monitoring" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Set up private probes A private probe is a [probe](/docs/grafana-cloud/testing/synthetic-monitoring/introduction/#probes) that you install in your environment or another location you choose. It runs tests against your targets by sending requests and recording the responses. Unlike [public probes](/docs/grafana-cloud/testing/synthetic-monitoring/create-checks/public-probes/) operated by Grafana Labs, private probes are only accessible to you and only write data to your Grafana Cloud account. Private probes are instances of the open source Grafana [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent). ## Before you begin > Note > > By default, private probes collect and report anonymous, non-personally identifiable information (PII) about the running probe to a remote statistics server operated by Grafana Labs. Synthetic Monitoring maintainers use that data to understand how the open source community operates private probes. > > To opt out, use the CLI flag `-disable-usage-reports=true`. Refer to [Anonymous usage statistics reporting](#anonymous-usage-statistics-reporting) for more information on what’s collected. - You must have the *Admin* role in Grafana to access the Synthetic Monitoring config page. - [Install and configure](/docs/grafana-cloud/testing/synthetic-monitoring/set-up/grafana-cloud/) Synthetic Monitoring before adding a private probe. - Installing from the Grafana APT repository on amd64-based Debian and Ubuntu systems is the only packaging option provided. On other systems, you need to build the agent from [source](https://www.github.com/grafana/synthetic-monitoring-agent). - You can install multiple private probes. Each private probe has its own unique **Probe Authentication Token**. ## Add a new probe in your Grafana instance 1. Navigate to **Testing & synthetics > Synthetics > Probes**. 2. Click **Add Private Probe**. 3. Enter a **Probe Name**. 4. Enter a **Latitude** and **Longitude** to see an accurate location on the map panel. 5. Enter a **Region**. Defaults are `AMER`, `EMEA`, and `APAC`, but you may define your own. 6. Optionally enter up to three custom labels to identify your probe. Probe labels can’t use reserved system label names. Refer to [System labels](/docs/grafana-cloud/testing/synthetic-monitoring/system-labels/). 7. Click **Save**. 8. Copy the **Probe Authentication Token** and save it for the next step. ## Probe API Server URL Based on the region of your stack, you need to use a different API server URL when setting up a private probe. If you are unsure about the region of your stack, go to your Grafana instance and the **Config** page under **Synthetic Monitoring** and search for `backend address`. > Note > > The config expects API server URLs without `https://`. If you receive an error, verify that you have omitted `https://`. Expand table | Region | Provider | API server URL | Backend address | Secrets proxy | |--------------------|----------|------------------------------------------------------------|-------------------------------------------------------|--------------------------------------------------------------------------------------| | Amsterdam | Azure | `synthetic-monitoring-grpc-eu-west-3.grafana.net:443` | `synthetic-monitoring-api-eu-west-3.grafana.net` | `gsm-proxy-prod-eu-west-3.grafana.net` | | Australia | GCP | `synthetic-monitoring-grpc-au-southeast.grafana.net:443` | `synthetic-monitoring-api-au-southeast.grafana.net` | `gsm-proxy-prod-au-southeast-0.grafana.net` | | Australia | AWS | `synthetic-monitoring-grpc-au-southeast-1.grafana.net:443` | `synthetic-monitoring-api-au-southeast-1.grafana.net` | `gsm-proxy-prod-au-southeast-1.grafana.net` | | Brazil | GCP | `synthetic-monitoring-grpc-sa-east-0.grafana.net:443` | `synthetic-monitoring-api-sa-east-0.grafana.net` | `gsm-proxy-prod-sa-east-0.grafana.net` | | Brazil | AWS | `synthetic-monitoring-grpc-sa-east-1.grafana.net:443` | `synthetic-monitoring-api-sa-east-1.grafana.net` | `gsm-proxy-prod-sa-east-1.grafana.net` | | Canada | AWS | `synthetic-monitoring-grpc-ca-east-0.grafana.net:443` | `synthetic-monitoring-api-ca-east-0.grafana.net` | `gsm-proxy-prod-ca-east-0.grafana.net` | | Europe | AWS | `synthetic-monitoring-grpc-eu-west-2.grafana.net:443` | `synthetic-monitoring-api-eu-west-2.grafana.net` | `gsm-proxy-prod-eu-west-2.grafana.net`, `gsm-proxy-prod-eu-west-4.grafana.net` | | Europe | AWS | `synthetic-monitoring-grpc-eu-west-6.grafana.net:443` | `synthetic-monitoring-api-eu-west-6.grafana.net` | `gsm-proxy-prod-eu-west-6.grafana.net` | | Europe | GCP | `synthetic-monitoring-grpc-eu-west.grafana.net:443` | `synthetic-monitoring-api-eu-west.grafana.net` | `gsm-proxy-prod-eu-west-0.grafana.net` | | India | GCP | `synthetic-monitoring-grpc-ap-south-0.grafana.net:443` | `synthetic-monitoring-api-ap-south-0.grafana.net` | `gsm-proxy-prod-ap-south-0.grafana.net` | | India | AWS | `synthetic-monitoring-grpc-ap-south-1.grafana.net:443` | `synthetic-monitoring-api-ap-south-1.grafana.net` | `gsm-proxy-prod-ap-south-1.grafana.net` | | Japan | AWS | `synthetic-monitoring-grpc-ap-northeast-0.grafana.net:443` | `synthetic-monitoring-api-ap-northeast-0.grafana.net` | `gsm-proxy-prod-ap-northeast-0.grafana.net` | | Saudi Arabia | GCP | `synthetic-monitoring-grpc-me-central-0.grafana.net:443` | `synthetic-monitoring-api-me-central-0.grafana.net` | `gsm-proxy-prod-me-central-0.grafana.net` | | Singapore | GCP | `synthetic-monitoring-grpc-ap-southeast-0.grafana.net:443` | `synthetic-monitoring-api-ap-southeast-0.grafana.net` | `gsm-proxy-prod-ap-southeast-0.grafana.net` | | Singapore | AWS | `synthetic-monitoring-grpc-ap-southeast-1.grafana.net:443` | `synthetic-monitoring-api-ap-southeast-1.grafana.net` | `gsm-proxy-prod-ap-southeast-1.grafana.net` | | Indonesia | AWS | `synthetic-monitoring-grpc-ap-southeast-2.grafana.net:443` | `synthetic-monitoring-api-ap-southeast-2.grafana.net` | `gsm-proxy-prod-ap-southeast-2.grafana.net` | | Sweden | AWS | `synthetic-monitoring-grpc-eu-north-0.grafana.net:443` | `synthetic-monitoring-api-eu-north-0.grafana.net` | `gsm-proxy-prod-eu-north-0.grafana.net` | | Switzerland | AWS | `synthetic-monitoring-grpc-eu-central-0.grafana.net:443` | `synthetic-monitoring-api-eu-central-0.grafana.net` | `gsm-proxy-prod-eu-central-0.grafana.net` | | UAE | AWS | `synthetic-monitoring-grpc-me-central-1.grafana.net:443` | `synthetic-monitoring-api-me-central-1.grafana.net` | `gsm-proxy-prod-me-central-1.grafana.net` | | UK | GCP | `synthetic-monitoring-grpc-gb-south.grafana.net:443` | `synthetic-monitoring-api-gb-south.grafana.net` | `gsm-proxy-prod-gb-south-0.grafana.net` | | UK | AWS | `synthetic-monitoring-grpc-gb-south-1.grafana.net:443` | `synthetic-monitoring-api-gb-south-1.grafana.net` | `gsm-proxy-prod-gb-south-1.grafana.net` | | United States East | AWS | `synthetic-monitoring-grpc-us-east-0.grafana.net:443` | `synthetic-monitoring-api-us-east-0.grafana.net` | `gsm-proxy-prod-us-east-0.grafana.net`, `gsm-proxy-prod-us-east-2.grafana.net` | | United States East | AWS | `synthetic-monitoring-grpc-us-east-3.grafana.net:443` | `synthetic-monitoring-api-us-east-3.grafana.net` | `gsm-proxy-prod-us-east-3.grafana.net` | | United States East | GCP | `synthetic-monitoring-grpc-us-east-1.grafana.net:443` | `synthetic-monitoring-api-us-east-1.grafana.net` | `gsm-proxy-prod-us-east-1.grafana.net` | | United States West | AWS | `synthetic-monitoring-grpc-us-west-0.grafana.net:443` | `synthetic-monitoring-api-us-west-0.grafana.net` | `gsm-proxy-prod-us-west-0.grafana.net` | | United States | Azure | `synthetic-monitoring-grpc-us-central-7.grafana.net:443` | `synthetic-monitoring-api-us-central-7.grafana.net` | `gsm-proxy-prod-us-central-7.grafana.net` | | United States | GCP | `synthetic-monitoring-grpc.grafana.net:443` | `synthetic-monitoring-api.grafana.net` | `gsm-proxy-prod-us-central-0.grafana.net`, `gsm-proxy-prod-us-central-4.grafana.net` | ## Firewall configuration Private probes initiate connections to Synthetic Monitoring services and monitored targets. Configure the probe host’s firewall to allow the outbound connections in the following sections. If a monitored target has its own firewall, configure that firewall separately to allow inbound connections from the probe host. ### Synthetic Monitoring services - Allow outbound connections from the probe host to the “API server URL” specified in the preceding table. - Allow outbound connections from the probe host to the “Secrets proxy” specified in the preceding table. ### Prometheus and Loki instances Allow outbound connections from the probe host to the Prometheus server URL and the Loki URL: 1. Log in to Grafana Cloud and open your stack. 2. Navigate to **Home** > **Testing & synthetics** > **Synthetics** > [**Config**](/launch/a/grafana-synthetic-monitoring-app/config). 3. Under the **General** tab, scroll down to **Data sources**. 4. For the Prometheus server URL: 1. Click the Prometheus data source. For example: `grafanacloud-STACK_NAME-prom`. 2. Copy the **Prometheus server URL** under the **Settings** tab -> **Connection**. 5. For the Loki URL: 1. Click the Loki data source. For example: `grafanacloud-STACK_NAME-logs`. 2. Copy the **URL** under the **Settings** tab -> **Connection**. 6. Update the probe host’s firewall rules with both URLs. ### External script imports - If you use k6 scripts that import external modules, for example, k6 modules from `jslib.k6.io`, allow outbound connections from the probe host to the module hosts. > Note > > MultiHTTP test executions make outbound connections to `jslib.k6.io` when run. If you run MultiHTTP checks on private probes behind a firewall, allow outbound connections from the probe host to `jslib.k6.io`. ### Monitored targets Allow outbound connections from the probe host to each monitored target. The required protocol depends on the check type. For example: - ICMP: for ping and traceroute checks. - UDP or TCP: for DNS checks. - TCP: for HTTP and MultiHTTP checks. If a target’s firewall blocks these connections, configure the target’s firewall, not the probe host’s firewall, to allow inbound connections from the probe host over the required protocol. ## Install the agent on Ubuntu/Debian You can install the agent using the Grafana official APT repository. 1. Add the required packages and the repository GPG key. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo apt install -y wget gnupg sudo mkdir -p /etc/apt/keyrings/ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg ``` 2. Add the repository for stable releases. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list ``` 3. Run the following commands. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo apt update sudo apt install synthetic-monitoring-agent ``` 4. Configure the probe. 1. Open `/etc/synthetic-monitoring/synthetic-monitoring-agent.conf`. 2. Replace `` with the probe authentication token. 3. Replace `API_SERVER` with Probe API Server URL based on your stack region. (defaults to US region) 4. Save the file. 5. Restart the agent. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo service synthetic-monitoring-agent restart ``` The new probe appears on your **Synthetics > Probes** page. ## Install the agent on distributions using yum You can install the agent using the official YUM repository. 1. Add the repository and the GPG key to the configuration. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo yum-config-manager --add-repo https://rpm.grafana.com wget https://rpm.grafana.com/gpg.key sudo rpm --import gpg.key ``` 2. Install the agent. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo yum install synthetic-monitoring-agent sudo systemctl status synthetic-monitoring-agent.service ``` This last command will tell you that the service is loaded but disabled. Before enabling, you need to [add the probe to your Synthetic Monitoring configuration](#add-a-new-probe-in-your-grafana-instance). 3. Configure the agent. Edit the file `/etc/synthetic-monitoring/synthetic-monitoring-agent.conf` and make the following changes: - Replace `` with the probe authentication token. - Replace `API_SERVER` with the API Server URL corresponding to your stack’s region. Defaults to the US region. 4. Enable the service and start it. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo systemctl enable synthetic-monitoring-agent.service sudo systemctl start synthetic-monitoring-agent.service ``` ## Deploy the agent using Docker You can find the [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent) Docker image on [Docker Hub](https://hub.docker.com/r/grafana/synthetic-monitoring-agent) 1. Pull image from Docker Hub. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash docker pull grafana/synthetic-monitoring-agent:latest ``` 2. Export required configuration as environment variables. Update the placeholders with your probe authentication token and the Probe API Server URL for your stack region. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash export API_TOKEN=@@@SM_PROBE_TOKEN@@@ export API_SERVER="@@@SM_PROBE_API_SERVER_URL@@@" ``` 3. Start the agent. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash docker run grafana/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --verbose=true ``` The new probe appears on your **Synthetics > Probes** page. ## Deployment with Kubernetes You can deploy the [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent) to Kubernetes using the Synthetic Monitoring Agent Docker image. In the agent GitHub repository, you can also find a [Kubernetes Deployment example](https://github.com/grafana/synthetic-monitoring-agent/tree/main/examples/kubernetes). The following section uses the example files to deploy a Synthetic Monitoring agent in a Kubernetes namespace. 1. Get the deployment files. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/namespace.yaml wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/secret.yaml wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/deployment.yaml ``` 2. Create the `synthetic-monitoring` namespace. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f namespace.yaml ``` 3. Create a secret with the Probe Authentication Token. > Replace `YOUR_TOKEN_HERE` in `secret.yaml` with your Probe Authentication Token. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f secret.yaml # verify secret kubectl get secrets -n synthetic-monitoring ``` 4. Deploy the Synthetic Monitoring Agent. > Replace `PROBE_API_SERVER_URL` in `deployment.yaml` with Probe API Server URL based on your stack region. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f deployment.yaml ``` The new probe appears on your **Synthetics > Probes** page. ## Calculate resource requirements Resources used by private probes may vary based on the targets being monitored, so it can be hard to predict them in advance. Below, we provide a rough estimate to get you started. The baseline CPU and memory resources, if you want to run: - API endpoint checks: `50m`/`64Mi`. - The above, plus multi-step and/or scripted checks: `200m`/`256Mi`. - The above, plus browser checks: `1000m`/`1024Mi`. Then, for each check per minute of the following types, add: - API endpoint (HTTP, Ping, DNS, TCP, Traceroute): `5m`/`2Mi`. - Multi-step and/or scripted: `20m`/`32Mi` - Browser checks: `100m`/`256Mi`. As mentioned above, keep in mind that these are ballpark estimates. Monitor the resource usage of your probe, and stay on the lookout for CPU throttling and OOM events to see if you need to adjust any resources. ## Configure feature flags You can use feature flags to enable features in the private probe. There are two ways to enable features: - Edit the file `/etc/synthetic-monitoring/synthetic-monitoring-agent.conf` and make the following changes: ini ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```ini FEATURES = ``` - Export the list of features as an environment variable. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash export FEATURES= ``` ### Available feature flags These features are disabled by default. Expand table | Feature flag name | Description | |-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `adhoc` | Click the **Test** button in the check editor UI to run checks out of schedule. If this isn’t enabled, the test result is a timeout. | | `k6` | Run k6-based checks. | | `traceroute` | Run `traceroute` checks, in addition to `ping` checks. Depending on your environment, you might need to grant specific permission to the probe process or container. | ## Run k6 checks targeting private IP addresses By default, private probes block k6 request targets in the 10.0.0.0/8 CIDR range. You can override this behavior by setting the CLI flag `--blocked-nets=""`. - When running Docker, append `--blocked-nets=""` to your `docker run` command. - On Linux installations, run `systemctl edit synthetic-monitoring-agent` and append `--blocked-nets=""` to `ExecStart=`. ## Running k6 browser checks on a private probe To be able to run [k6 browser checks](/docs/grafana-cloud/testing/synthetic-monitoring/create-checks/checks/k6-browser/) on private probes, you will need to have a Chromium-compatible web browser available on the system where the probe is running. Additionally, you need to make sure that in the probe settings page in your Grafana stack (Testing & synthetics > Synthetics > Probes), both “Disable scripted checks” and “Disable browser checks” are *unchecked*. ### k6 browser checks on Ubuntu, Debian, and RPM-based distributions If you installed the [Synthetic Monitoring Agent](https://www.github.com/grafana/synthetic-monitoring-agent) on your distribution via your package manager, your distribution needs to have Chromium installed, and a `chromium` binary available in `$PATH`. Alternatively, if you do not want to make `chromium` widely available, or you need to use a specific installation, you can add set `K6_BROWSER_EXECUTABLE_PATH` in the synthetic monitoring agent’s environment to the full path of a `chromium` binary. For advanced use cases, you can also define `K6_BROWSER_ARGS` with additional arguments that the agent will use when launching Chromium. ### k6 browser checks on Docker, Kubernetes, and other container-based environments For container-based environments, we provide a special image that already includes an installation of Chromium. These images are tagged with the `-browser` suffix, for example `grafana/synthetic-monitoring-agent:vX.Y.Z-browser`, as opposed to `grafana/synthetic-monitoring-agent:vX.Y.Z`. Agent images tagged *with* the `-browser` suffix can run all types of checks, while those tagged *without* the `-browser` suffix can run every check type except k6 browser checks. If you do not plan to use browser checks, we recommend using the standard image, as it is lighter and more minimal. Swapping to a `-browser` image can be done at any time without any manual intervention, other than changing the tag. ## Anonymous usage statistics reporting > Note > > The anonymous usage statistics is *enabled by default*. You can opt out by setting the CLI flag `-disable-usage-reports=true`. ### The statistics server When usage statistics reporting is enabled, information is collected by a server that Grafana Labs operates. Statistics are sent to `https://stats.grafana.org`. ### Which information is collected When the usage statistics reporting is enabled, probes collect the following information: - Information about the **probe**, including: - A unique hash representation of the probe - The probe version, such as `0.39.0` - The installation source, such as `docker` - The probe id, such as `1100` - The tenant id, such as `1100` - The features enabled, such as `k6` - Information about the **environment**, including: - The operating system, such as `linux` - The architecture, such as `amd64` - The os details, such as `Ubuntu 24.04.2 LTS` ## Keep the agent up to date New Synthetic Monitoring features are delivered through Synthetic Monitoring Agent releases, so probes running older agent versions don’t support them. Upgrade your private probes regularly to stay on a supported version. Keeping the agent up to date is especially important for the [label migration](/docs/grafana-cloud/testing/synthetic-monitoring/migrate-labels/): the un-prefixed label behavior is delivered by the agent. Upgrade all your private probes to the latest agent version *before* enabling dual-write. > Warning > > If some of your probes run an older agent version when you migrate, those probes keep emitting custom labels with the `label_` prefix while upgraded probes emit the new format. This leaves the same check with inconsistent labels across probes. > > This includes probes that are offline or that you no longer use: upgrade them or delete them before you migrate. ## Troubleshooting private probes If you are facing problems with your private probes, check the following: 1. Check the probe version using the `-version` flag and make sure you are running a supported version. 2. Make sure the probe has a valid probe access token. 3. Make sure the probe can connect to the Synthetic Monitoring API server. 4. Run the Synthetic Monitoring Agent in debug mode with the `-debug` and `-verbose` flags and check for errors in the logs. 5. Check the Synthetic Monitoring Agent metrics. The default address for Synthetic Monitoring Agent Prometheus metrics is `localhost:4050/metrics`. If you are still facing issues with Private Probes, ask for help in the [community forum](https://community.grafana.com/). Refer to [Account types and support available for Grafana Cloud users](/docs/grafana-cloud/account-management/support/#account-types-and-support-available-for-grafana-cloud-users) for more information. ## Production deployments The agent process exposes Prometheus-style metrics on an HTTP server running on port 4050 with the “/metrics” endpoint. You can scrape and monitor your private probe using Prometheus or [Grafana Alloy](https://github.com/grafana/alloy). ## Next steps You must reconfigure any [checks](/docs/grafana-cloud/testing/synthetic-monitoring/create-checks/checks/) to use your new probe even if you selected all probes when initially creating the check. The Synthetic Monitoring Agent is open source and [available on GitHub](https://www.github.com/grafana/synthetic-monitoring-agent).