Documentation Grafana Cloud Observe and act Test and validate Synthetic Monitoring Analyze results Troubleshoot a failing check

Troubleshoot a failing check

Use this procedure when a Synthetic Monitoring check reports failures or has missing results. Start with one execution, find the operation that failed, and compare the result across probes before deciding which system or configuration to investigate.

Before you begin

• Identify the affected check and the approximate time the problem occurred, including the time zone.
• You need access to the check’s dashboard and execution logs. If you can’t open them, ask your Grafana administrator for access.
• To investigate a network or application error, you need access to the relevant application, server, or network logs, or help from the team that manages them.

Open an affected execution

1. Go to Testing & synthetics > Synthetics > Checks.
2. Use Search checks to find the check by its job name or the endpoint you’re monitoring, then click View dashboard.
3. Use the time picker to select the period when the problem occurred. Check the displayed time zone, and set the probe filter to All so you can compare locations.
4. Below the summary panels, find the time point explorer and select Reachability to view individual probe executions.
5. Click a time point in the graph near the time of the problem. Use the mini map above the graph to move to an earlier part of the selected period if necessary.
6. In the details below the graph, select the affected probe’s tab.

The details show either execution logs or a message about the execution’s state. The left and right arrows above the probe tabs are Previous timepoint and Next timepoint. Use them to move between intervals. Choose your next step from the state:

The newest time point can still be waiting for results. Don’t use that message alone to conclude that the check or probe is broken. For the explorer’s controls and state definitions, refer to Visualize check execution.

Find the operation that failed

1. Keep Event selected in the execution details to read the parsed log messages.
2. Record the Execution ID, probe name, and execution timestamps.
3. Find an error message that describes a specific operation. A final message such as Check failed or script did not execute successfully summarizes the result; read the preceding messages to find the underlying error.
4. Record the error text and any associated err, error, filename, or line fields. For an HTTP request, also record the requested host and resolved IP address if the logs show them.

Use the message to select an investigation:

For example, an HTTP execution can contain this sequence, with fields omitted for clarity:

text Copy

Beginning check                 timeout_seconds=60
Resolved target address         ip=34.133.192.185
Making HTTP request             host=quickpizza.grafana.fun
Error for HTTP request          err=... context deadline exceeded
Check failed                    duration_seconds=60.00089016

This execution resolved the hostname, then timed out during the HTTP request. It provides a resolved address, operation, and time window for the application or network team to investigate. It doesn’t show that increasing the check timeout would fix the problem.

A browser failure can instead identify a script line and an unmet expectation:

text Copy

test aborted: error=expect(locator).toBeVisible()
filename=script.k6 line=48 expected=visible received=hidden

Start with the locator and the action leading to line 48. Reaching the application doesn’t establish that the expected page element appeared.

Compare probe executions

1. Keep the affected time point selected and record the failing probe’s operation and error.
2. Select another probe’s tab in the same time point. Record whether it succeeded, failed, or has no reported result. If it failed, identify its operation and error using the same steps.
3. Repeat for the remaining probes. Compare the resolved addresses as well as the error messages when those details are available.
4. If no probe succeeded, use Previous timepoint to look for an earlier successful execution from the same probe. Record both timestamps so you can identify what changed between them. If the selected period has no successful executions, continue with the failure evidence you have.

Use the comparison to decide whom to involve and what evidence to provide:

• Some locations succeed and others fail: Ask the team that manages the application’s network or security rules to search firewall, web application firewall (WAF), or content delivery network (CDN) logs for the affected time and public probe source addresses. Use the current Synthetic Monitoring source IP ranges. Check for a matching denial or challenge before changing an access rule. For private probes, use the source address the application or server sees from that probe’s network. A successful request from your laptop doesn’t test the same network path.
• Locations fail at the same operation: Send the operation, resolved addresses, timestamps, and errors to the application or network team. Ask them to check the corresponding listener, request, or access rule. Matching errors narrow the investigation but don’t prove a shared cause.
• A script or assertion starts failing after a change: Compare the failing line and expected result with the application change and the check’s configuration. If you see a check-update annotation, record its timestamp and ask the check owner to review what changed. Don’t weaken the assertion solely to make the check succeed.

The dashboard’s Uptime and Reachability panels help summarize this scope. High uptime with lower reachability can reflect successes from some probes and failures from others. With one probe, the two indicators don’t provide a comparison between locations. For calculations and examples, refer to Uptime and reachability.

Inspect a complete execution in Explore

Use Explore when you need to filter the logs or share a reproducible query:

1. Select the relevant time point and probe in the execution details, then click View Logs in Explore, the log icon beside the time point navigation arrows. Explore opens with the log data source, check labels, probe, and time range populated.

2. Compare Explore’s time range with the execution’s start and finish timestamps. If the execution finishes in the next time point, widen the range to include its finish. Use Zoom out time range, or set a range with the time picker.

3. In Code, keep the generated job, instance, and probe filters. If the logs expose an execution_id field, append a filter for the Execution ID you recorded:

   logql Copy

   {job="YOUR_JOB", instance="YOUR_INSTANCE", probe="YOUR_PROBE"}
     | logfmt
     | execution_id="YOUR_EXECUTION_ID"

4. Copy the job, instance, and probe values from the generated query, replace YOUR_EXECUTION_ID with the ID you recorded, and click Run query. Keep the generated instance value even if it isn’t a URL; scripted and browser checks can use a descriptive value.

5. Confirm that the returned entries include the execution’s beginning and final result. If they don’t, check the time range and filter values before concluding that logs are missing. If execution_id isn’t available, retain the generated label filters and use the timestamps to identify the relevant entries.

Executions belong to the time point in which they start, but their final log messages can be timestamped in a later interval. A query restricted to the starting interval can therefore show an incomplete execution or messages from a preceding execution.

Investigate missing results

A missing result doesn’t report whether the endpoint you’re monitoring was available. Investigate the data gap separately from a reported failure:

1. Use Previous timepoint to select an earlier interval that should have finished. If the earlier execution has results, distinguish that completed execution from the recent one still waiting for results.
2. Check the selected period against the check’s creation time and any no-data annotations. Move to a period after creation and within the available retention period if necessary.
3. Select another probe’s tab to find whether the gap affects one probe or several.
4. If logs are available, follow Inspect a complete execution in Explore to check the full execution interval. A gap in one view doesn’t establish that both logs and metrics are absent.
5. If other checks use the affected probe, open their dashboards from Checks, select the same period, and inspect that probe’s results. Missing results across several checks can indicate a wider probe or reporting issue rather than a gap limited to one check.
6. If completed intervals still have no results, collect the evidence for your Grafana administrator or Grafana support. Include which probes and data are missing. The execution state alone doesn’t identify the cause, so don’t reset the plugin or change data sources based only on that state.

Uptime excludes time points with no reported results. A high uptime value therefore doesn’t establish availability during a gap. For an example of interrupted result reporting, refer to A visual example and known limitations.

Verify recovery or escalate

After addressing the suspected cause, return to the check dashboard and select a time range after the change. Keep probe set to All, then inspect new completed executions from each previously affected probe. Confirm that the intended request or journey succeeds under the check’s success criteria.

If you need another team or Grafana support to continue the investigation, provide:

• The check’s job name, monitored endpoint, and check type.
• The affected time range and time zone.
• The failing and successful probes, with Execution ID values where available.
• The specific operation, error message, and script line or resolved endpoint IP address where applicable.
• Relevant log entries or an Explore query with the full execution interval.
• Any check or application changes near the start of the problem.
• For a data gap, which probes have missing results, whether logs, metrics, or both are affected, and whether other checks using those probes have the same gap.