https://grafana.com/oss/prometheus/exporters/node-exporter/Recent content in Node exporter on Grafana LabsHugo -- gohugo.ioenhttps://grafana.comMon, 01 Jan 0001 00:00:00 +0000https://grafana.com<p>This quickstart includes the following alerting rules:</p> <ul> <li> <p><code>NodeFilesystemSpaceFillingUp</code></p> <p>Filesystem on device at instace has only X available space left and is filling up.</p> </li> <li> <p><code>NodeFilesystemAlmostOutOfSpace</code></p> <p>Filesystem on device at instance has only X available space left.</p> </li> <li> <p><code>NodeFilesystemFilesFillingUp</code></p> <p>Filesystem on device at instance has only X available inodes left and is filling up.</p> </li> <li> <p><code>NodeFilesystemAlmostOutOfFiles</code></p> <p>Filesystem on device at instance has only X available inodes left</p> </li> <li> <p><code>NodeNetworkReceiveErrs</code></p> <p>Instance interface device has encountered X receive errors in the last two minutes.</p> </li> <li> <p><code>NodeHighNumberConntrackEntriesUsed</code></p> <p>% of conntrack entries are used.</p> </li> <li> <p><code>NodeTextFileCollectorScrapeError</code></p> <p>Node Exporter text file collector failed to scrape.</p> </li> <li> <p><code>NodeClockSkewDetected</code></p> <p>Clock on instance is out of sync by more than 300s. Ensure NTP is configured correctly on this host.</p> </li> <li> <p><code>NodeClockNotSynchronising</code></p> <p>Clock on instance is not synchronising. Ensure NTP is configured on this host.</p> </li> <li> <p><code>NodeRAIDDegraded</code></p> <p>RAID array &lsquo;device&rsquo; on instance is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically.</p> </li> <li> <p><code>NodeRAIDDiskFailure</code></p> <p>At least one device in RAID array on instance failed. Array &lsquo;device&rsquo; needs attention and possibly a disk swap.</p> </li> </ul> <a class="btn btn--primary br-5 mb-1" href="assets/node_alerts.yaml" rel="noopener noreferrer">Download the following alerting rules YAML file</a> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">YAML</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-yaml">&#34;groups&#34;: - &#34;name&#34;: &#34;node-exporter&#34; &#34;rules&#34;: - &#34;alert&#34;: &#34;NodeFilesystemSpaceFillingUp&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available space left and is filling up.&#34; &#34;summary&#34;: &#34;Filesystem is predicted to run out of space within the next 24 hours.&#34; &#34;expr&#34;: | ( node_filesystem_avail_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_size_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 40 and predict_linear(node_filesystem_avail_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;}[6h], 24*60*60) &lt; 0 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeFilesystemSpaceFillingUp&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available space left and is filling up fast.&#34; &#34;summary&#34;: &#34;Filesystem is predicted to run out of space within the next 4 hours.&#34; &#34;expr&#34;: | ( node_filesystem_avail_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_size_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 20 and predict_linear(node_filesystem_avail_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;}[6h], 4*60*60) &lt; 0 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;critical&#34; - &#34;alert&#34;: &#34;NodeFilesystemAlmostOutOfSpace&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available space left.&#34; &#34;summary&#34;: &#34;Filesystem has less than 5% space left.&#34; &#34;expr&#34;: | ( node_filesystem_avail_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_size_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 5 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeFilesystemAlmostOutOfSpace&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available space left.&#34; &#34;summary&#34;: &#34;Filesystem has less than 3% space left.&#34; &#34;expr&#34;: | ( node_filesystem_avail_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_size_bytes{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 3 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;critical&#34; - &#34;alert&#34;: &#34;NodeFilesystemFilesFillingUp&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available inodes left and is filling up.&#34; &#34;summary&#34;: &#34;Filesystem is predicted to run out of inodes within the next 24 hours.&#34; &#34;expr&#34;: | ( node_filesystem_files_free{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_files{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 40 and predict_linear(node_filesystem_files_free{job=&#34;node&#34;,fstype!=&#34;&#34;}[6h], 24*60*60) &lt; 0 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeFilesystemFilesFillingUp&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available inodes left and is filling up fast.&#34; &#34;summary&#34;: &#34;Filesystem is predicted to run out of inodes within the next 4 hours.&#34; &#34;expr&#34;: | ( node_filesystem_files_free{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_files{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 20 and predict_linear(node_filesystem_files_free{job=&#34;node&#34;,fstype!=&#34;&#34;}[6h], 4*60*60) &lt; 0 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;critical&#34; - &#34;alert&#34;: &#34;NodeFilesystemAlmostOutOfFiles&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available inodes left.&#34; &#34;summary&#34;: &#34;Filesystem has less than 5% inodes left.&#34; &#34;expr&#34;: | ( node_filesystem_files_free{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_files{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 5 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeFilesystemAlmostOutOfFiles&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \&#34;%.2f\&#34; $value }}% available inodes left.&#34; &#34;summary&#34;: &#34;Filesystem has less than 3% inodes left.&#34; &#34;expr&#34;: | ( node_filesystem_files_free{job=&#34;node&#34;,fstype!=&#34;&#34;} / node_filesystem_files{job=&#34;node&#34;,fstype!=&#34;&#34;} * 100 &lt; 3 and node_filesystem_readonly{job=&#34;node&#34;,fstype!=&#34;&#34;} == 0 ) &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;critical&#34; - &#34;alert&#34;: &#34;NodeNetworkReceiveErrs&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \&#34;%.0f\&#34; $value }} receive errors in the last two minutes.&#34; &#34;summary&#34;: &#34;Network interface is reporting many receive errors.&#34; &#34;expr&#34;: | rate(node_network_receive_errs_total[2m]) / rate(node_network_receive_packets_total[2m]) &gt; 0.01 &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeNetworkTransmitErrs&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \&#34;%.0f\&#34; $value }} transmit errors in the last two minutes.&#34; &#34;summary&#34;: &#34;Network interface is reporting many transmit errors.&#34; &#34;expr&#34;: | rate(node_network_transmit_errs_total[2m]) / rate(node_network_transmit_packets_total[2m]) &gt; 0.01 &#34;for&#34;: &#34;1h&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeHighNumberConntrackEntriesUsed&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;{{ $value | humanizePercentage }} of conntrack entries are used.&#34; &#34;summary&#34;: &#34;Number of conntrack are getting close to the limit.&#34; &#34;expr&#34;: | (node_nf_conntrack_entries / node_nf_conntrack_entries_limit) &gt; 0.75 &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeTextFileCollectorScrapeError&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Node Exporter text file collector failed to scrape.&#34; &#34;summary&#34;: &#34;Node Exporter text file collector failed to scrape.&#34; &#34;expr&#34;: | node_textfile_scrape_error{job=&#34;node&#34;} == 1 &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeClockSkewDetected&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Clock on {{ $labels.instance }} is out of sync by more than 300s. Ensure NTP is configured correctly on this host.&#34; &#34;summary&#34;: &#34;Clock skew detected.&#34; &#34;expr&#34;: | ( node_timex_offset_seconds &gt; 0.05 and deriv(node_timex_offset_seconds[5m]) &gt;= 0 ) or ( node_timex_offset_seconds &lt; -0.05 and deriv(node_timex_offset_seconds[5m]) &lt;= 0 ) &#34;for&#34;: &#34;10m&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeClockNotSynchronising&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;Clock on {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.&#34; &#34;summary&#34;: &#34;Clock not synchronising.&#34; &#34;expr&#34;: | min_over_time(node_timex_sync_status[5m]) == 0 and node_timex_maxerror_seconds &gt;= 16 &#34;for&#34;: &#34;10m&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34; - &#34;alert&#34;: &#34;NodeRAIDDegraded&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;RAID array &#39;{{ $labels.device }}&#39; on {{ $labels.instance }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically.&#34; &#34;summary&#34;: &#34;RAID Array is degraded&#34; &#34;expr&#34;: | node_md_disks_required - ignoring (state) (node_md_disks{state=&#34;active&#34;}) &gt; 0 &#34;for&#34;: &#34;15m&#34; &#34;labels&#34;: &#34;severity&#34;: &#34;critical&#34; - &#34;alert&#34;: &#34;NodeRAIDDiskFailure&#34; &#34;annotations&#34;: &#34;description&#34;: &#34;At least one device in RAID array on {{ $labels.instance }} failed. Array &#39;{{ $labels.device }}&#39; needs attention and possibly a disk swap.&#34; &#34;summary&#34;: &#34;Failed device in RAID array&#34; &#34;expr&#34;: | node_md_disks{state=&#34;failed&#34;} &gt; 0 &#34;labels&#34;: &#34;severity&#34;: &#34;warning&#34;</code></pre> </div> </div> <p>This alerting rule YAML file was generated using the Node-exporter <a href="https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin" target="_blank" rel="noopener noreferrer">mixin</a>. It uses the <code>job=node</code> label selector to query metrics by default. If you need to use a different selector, modify the selector in <a href="https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/config.libsonnet" target="_blank" rel="noopener noreferrer">config.libsonnet</a> and regenerate the dashboard following the instructions in the mixin repository.</p>https://grafana.comMon, 01 Jan 0001 00:00:00 +0000https://grafana.com <div class="bg-solitude p-1 mb-1" style=""> <p>Grafana Cloud comes with an ever-expanding set of integrations to quickly get an observability stack up and running in minutes. The Linux Node Integration, built-in to the Grafana Agent, exposes and scrapes essential system metrics and pushes them to Grafana Cloud. The agent will scrape metrics using an embedded Node Exporter, and Grafana Cloud will automatically provision tailored Grafana dashboards and alerts for visualizing and acting on this data.</p> <div> <a class="btn btn--primary br-5 " href="/auth/sign-up/create-user?pg=oss-prom-node-exporter&amp;plcmt=body-btn1-integration" rel="noopener noreferrer">Sign up for a free account to get started</a> </div> </div> <p>To learn more, check out the Grafana Cloud <a href="/docs/grafana-cloud/getting-started/gs-integrations/">docs</a>.</p> <h4 id="how-it-works">How it works</h4> <p>Configuring, installing, connecting, and maintaining Prometheus monitoring components typically involves significant domain knowledge. It can take quite a while to go from setup to dashboard and alerts. As the creators of Grafana - and core contributors to Prometheus and Cortex - we build simple integrations to abstract some of this work away in order to quickly get started. How it works:</p> <ol> <li>Sign up (or log in) for a <a href="/auth/sign-up/create-user?pg=oss-prom-node-exporter&amp;plcmt=body-txt-integration">free Grafana Cloud account</a>.</li> <li>Select the target you’d like to observe (an ever-expanding catalogue).</li> <li>Run a one-line command to install the Grafana Agent. The agent embeds and preconfigures Exporters to expose default metrics, and pushes them to the Grafana Cloud metrics backend.</li> <li>Voila! You’ll see tailored Grafana dashboards and will benefit from sane alerting defaults.</li> </ol> <p>Looking for a different Exporter or integration? Check out our <a href="/docs/grafana-cloud/integrations/">growing library of integrations for popular components</a> like MySQL, Postgres, Redis, Memcached and more.</p>https://grafana.comMon, 01 Jan 0001 00:00:00 +0000https://grafana.com<p><strong>Note: These dashboards depend on the recording rules in the previous tab. Be sure to import these before importing the dashboards.</strong></p> <h3 id="nodes-dashboard"><a href="/grafana/dashboards/13978">Nodes Dashboard</a></h3> <p><img class="lazyload d-inline-block" src="/api/dashboards/13978/images/9972/image" alt="Nodes Dashboard"/></p> <p>This dashboard includes panels for the following metrics:</p> <ul> <li>CPU Usage</li> <li>Load Average</li> <li>Memory Usage</li> <li>Disk I/O</li> <li>Disk Usage</li> <li>Network Received</li> <li>Network Transmitted</li> </ul> <p>You can import the dashboard directly using the following dashboard ID: <strong><code>13978</code></strong></p> <p>You can also download the dashboard JSON directly <a href="/api/dashboards/13978/revisions/1/download">here</a>.</p> <h3 id="use-method--node-dashboard"><a href="/grafana/dashboards/13977">USE Method / Node Dashboard</a></h3> <p><img class="lazyload d-inline-block" src="/api/dashboards/13977/images/9971/image" alt="USE Method / Node Dashboard"/></p> <p>This dashboard includes panels for the following metrics. To learn more about USE (Utilization, Saturation, and Errors) metrics, please see <a href="http://www.brendangregg.com/usemethod.html" target="_blank" rel="noopener noreferrer">The USE Method</a>:</p> <ul> <li>CPU Utilization</li> <li>CPU Saturation (Load per CPU)</li> <li>Memory Utilisation</li> <li>Memory Saturation (Major Page Faults)</li> <li>Net Utilization (Bytes Receive/Transmit)</li> <li>Net Saturation (Drops Receive/Transmit)</li> <li>Disk I/O Utilization</li> <li>Disk I/O Saturation</li> </ul> <p>You can import the dashboard directly using the following dashboard ID: <strong><code>13977</code></strong></p> <p>You can also download the dashboard JSON directly <a href="/api/dashboards/13978/revisions/1/download">here</a>.</p> <h3 id="use-method--cluster-dashboard"><a href="/grafana/dashboards/13971">USE Method / Cluster Dashboard</a></h3> <p><img class="lazyload d-inline-block" src="/api/dashboards/13971/images/9969/image" alt="USE Method / Cluster Dashboard"/></p> <p>This dashboard is similar to the USE Method / Node Dashboard but provides a cluster-level view of the same metrics. It includes panels for the following metrics:</p> <ul> <li>CPU Utilization</li> <li>CPU Saturation (Load per CPU)</li> <li>Memory Utilisation</li> <li>Memory Saturation (Major Page Faults)</li> <li>Net Utilization (Bytes Receive/Transmit)</li> <li>Net Saturation (Drops Receive/Transmit)</li> <li>Disk I/O Utilization</li> <li>Disk I/O Saturation</li> </ul> <p>You can import the dashboard directly using the following dashboard ID: <strong><code>13971</code></strong></p> <p>You can also download the dashboard JSON directly <a href="/api/dashboards/13978/revisions/1/download">here</a>.</p> <p>These dashboards were generated using the Node Exporter <a href="https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin" target="_blank" rel="noopener noreferrer">mixin</a>. They use the <code>job=node</code> label selector to query metrics by default. If you need to use a different selector, modify the selector in <a href="https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/config.libsonnet" target="_blank" rel="noopener noreferrer">config.libsonnet</a> and regenerate the dashboard following the instructions in the mixin repository.</p>https://grafana.comMon, 01 Jan 0001 00:00:00 +0000https://grafana.com<h2 id="node-exporter-quickstart">Node exporter quickstart</h2> <p>In this guide you&rsquo;ll learn how to set up and configure <a href="https://github.com/prometheus/node_exporter" target="_blank" rel="noopener noreferrer">Node Exporter</a> to collect Linux system metrics like CPU load and disk I/O and expose them as Prometheus-style metrics. You&rsquo;ll then configure <a href="https://prometheus.io/" target="_blank" rel="noopener noreferrer">Prometheus</a> to scrape Node Exporter metrics and optionally ship them to Grafana Cloud. Finally, you&rsquo;ll set up a preconfigured and curated set of <a href="https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/" target="_blank" rel="noopener noreferrer">recording rules</a>, Grafana <a href="/grafana/dashboards">dashboards</a>, and <a href="https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/" target="_blank" rel="noopener noreferrer">alerting rules</a>. At the end of this guide you&rsquo;ll have dashboards that you can use to visualize your Linux system metrics, and set of preconfigured alerts.</p> <p>If you&rsquo;re using <a href="/products/cloud/">Grafana Cloud</a>, the Linux Server Integration can help you get up and running quickly. The Linux Integration embeds Node Exporter into the <a href="https://github.com/grafana/agent" target="_blank" rel="noopener noreferrer">Grafana Agent</a> and automatically provisions alerting rules and dashboards, so you don&rsquo;t have to run through the steps in this guide. To learn how to set up Node Exporter using the Linux Server Integration, please see <a href="/docs/grafana-cloud/quickstart/agent_linuxnode/#monitoring-a-linux-host-using-the-linux-host-integration">Monitoring a Linux host using the Linux host integration</a> from the Grafana Cloud docs.</p> <h3 id="prerequisites">Prerequisites</h3> <p>Before you get started, you should have the following available to you:</p> <ul> <li>A Linux machine compatible with a Node Exporter release. To see a list of available releases, please see <a href="https://github.com/prometheus/node_exporter/releases/" target="_blank" rel="noopener noreferrer">Releases</a>.</li> <li>Prometheus running in your environment or directly on the Linux machine. To learn how to install Prometheus, please see <a href="https://prometheus.io/docs/prometheus/latest/installation/" target="_blank" rel="noopener noreferrer">Installation</a> from the Prometheus docs.</li> <li>Grafana running in your environment or directly on the Linux machine. To learn how to install Grafana, please see <a href="/docs/grafana/latest/installation/">Install Grafana</a> from the Grafana docs.</li> <li>(Optional) A Grafana Cloud account. Grafana Cloud hosts Grafana and a <a href="/oss/mimir/">Mimir</a>-based Prometheus metrics endpoint. You will still need to scrape metrics, using either Prometheus installed in your environment, or the <a href="https://github.com/grafana/agent" target="_blank" rel="noopener noreferrer">Grafana Agent</a>. To learn more about Grafana Cloud, please see <a href="/products/cloud/?pg=oss-prom-node-exporter&amp;plcmt=body-txt-installation">Grafana Cloud</a>.</li> </ul> <h3 id="step-1-setting-up-node-exporter">Step 1: Setting up Node Exporter</h3> <p>In this step you&rsquo;ll set up Node Exporter on your Linux machine to collect and expose system metrics.</p> <p>To begin, log in to your machine and download the relevant Node Exporter binary. In this guide we&rsquo;ll use <code>linux-amd64</code> but you should choose the one corresponding to your system&rsquo;s OS and architecture:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">wget https://github.com/prometheus/node_exporter/releases/download/v1.1.1/node_exporter-1.1.1.linux-amd64.tar.gz</code></pre> </div> </div> <p>Replace <code>1.1.1</code> with the version you&rsquo;d like to install. This guide may become stale so it&rsquo;s best to check the Node Exporter <a href="https://github.com/prometheus/node_exporter/releases/" target="_blank" rel="noopener noreferrer">Releases</a> page for the latest stable version.</p> <p>Unzip the tarball and <code>cd </code> into the directory:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">tar xvfz node_exporter-*.*-amd64.tar.gz cd node_exporter-*.*-amd64</code></pre> </div> </div> <p>Run the Node Exporter binary:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">./node_exporter</code></pre> </div> </div> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">level=info ts=2021-02-15T03:35:18.396Z caller=node_exporter.go:178 msg=&#34;Starting node_exporter&#34; version=&#34;(version=1.1.1, branch=HEAD, revision=4e837d4da79cc59ee3ed1471ba9a0d9547e95540)&#34; level=info ts=2021-02-15T03:35:18.396Z caller=node_exporter.go:179 msg=&#34;Build context&#34; build_context=&#34;(go=go1.15.8, user=root@7abcc101f1c2, date=20210213-13:30:54)&#34; level=warn ts=2021-02-15T03:35:18.396Z caller=node_exporter.go:181 msg=&#34;Node Exporter is running as root user. This exporter is designed to run as unpriviledged user, root is not required.&#34; level=info ts=2021-02-15T03:35:18.397Z caller=filesystem_common.go:74 collector=filesystem msg=&#34;Parsed flag --collector.filesystem.ignored-mount-points&#34; flag=^/(dev|proc|sys|var/lib/docker/.&#43;)($|/) level=info ts=2021-02-15T03:35:18.397Z caller=filesystem_common.go:76 collector=filesystem msg=&#34;Parsed flag --collector.filesystem.ignored-fs-types&#34; flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ level=info ts=2021-02-15T03:35:18.397Z caller=node_exporter.go:106 msg=&#34;Enabled collectors&#34; level=info ts=2021-02-15T03:35:18.397Z caller=node_exporter.go:113 collector=arp level=info ts=2021-02-15T03:35:18.397Z caller=node_exporter.go:113 collector=bcache . . . level=info ts=2021-02-15T03:38:01.468Z caller=node_exporter.go:113 collector=zfs level=info ts=2021-02-15T03:38:01.468Z caller=node_exporter.go:195 msg=&#34;Listening on&#34; address=:9100 level=info ts=2021-02-15T03:38:01.468Z caller=tls_config.go:191 msg=&#34;TLS is disabled.&#34; http2=false</code></pre> </div> </div> <p>If you see the above output, you successfully ran Node Exporter.</p> <p>Node Exporter publishes your system metrics in Prometheus format on port <code>9100</code>. You can test this using <code>curl</code>. You will need to open a new SSH session or background the Node Exporter process to use <code>curl</code>.</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">curl http://localhost:9100/metrics</code></pre> </div> </div> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">. . . process_virtual_memory_max_bytes 1.8446744073709552e&#43;19 ## HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler. ## TYPE promhttp_metric_handler_errors_total counter promhttp_metric_handler_errors_total{cause=&#34;encoding&#34;} 0 promhttp_metric_handler_errors_total{cause=&#34;gathering&#34;} 0 ## HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served. ## TYPE promhttp_metric_handler_requests_in_flight gauge promhttp_metric_handler_requests_in_flight 1 ## HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code. ## TYPE promhttp_metric_handler_requests_total counter promhttp_metric_handler_requests_total{code=&#34;200&#34;} 0 promhttp_metric_handler_requests_total{code=&#34;500&#34;} 0 promhttp_metric_handler_requests_total{code=&#34;503&#34;} 0</code></pre> </div> </div> <p>If you see the above output, you&rsquo;re ready to begin scraping Node Exporter metrics using Prometheus.</p> <p>To avoid running and managing Node Exporter from the command line, you can create a <code>systemd</code> service. To learn how to do this, please see <a href="/docs/grafana-cloud/agent/agent_as_service/">Creating a systemd service to manage the agent</a>. Replace the path to the agent binary with the path to Node Exporter.</p> <h3 id="step-2-scraping-node-exporter-using-prometheus">Step 2: Scraping Node Exporter using Prometheus</h3> <p>Now that Node Exporter is up and running on your machine, you can configure a Prometheus scrape job to collect and store Node Exporter metrics.</p> <p>Add the following scrape job config to the <code>scrape_configs</code> section of your <code>prometheus.yml</code> configuration file:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">- job_name: node static_configs: - targets: [&#39;linux_machine_IP_address:9100&#39;]</code></pre> </div> </div> <p>Replace <code>linux_machine_IP_address</code> with the IP address of the machine running Node Exporter. If you&rsquo;re running Prometheus on the same machine, this will be <code>localhost</code>. To learn more about configuring Prometheus, please see <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/" target="_blank" rel="noopener noreferrer">Configuration</a> from the Prometheus docs.</p> <p>If you don&rsquo;t have a <code>prometheus.yml</code> configuration file, create a simple one using your favorite text editor. Open your preferred text editor and paste in the following Prometheus configuration:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">global: scrape_interval: 15s scrape_configs: - job_name: node static_configs: - targets: [&#39;linux_machine_IP_address:9100&#39;]</code></pre> </div> </div> <p>This configuration tells Prometheus to scrape all jobs every 15 seconds. The only configured scrape job is called <code>node</code> and defines a <code>linux_machine_IP_address:9100</code> target. By default, Prometheus will scrape the <code>/metrics</code> endpoint using HTTP.</p> <p>Save and close the file. You can then run Prometheus with the file using the following command:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">./prometheus --config.file=./prometheus.yml</code></pre> </div> </div> <h4 id="shipping-metrics-to-grafana-cloud">Shipping metrics to Grafana Cloud</h4> <p>To ship Node Exporter metrics to Grafana Cloud from Prometheus, configure the <code>remote_write</code> parameter in your <code>prometheus.yml</code> configuration file. To learn more, please see <a href="/docs/grafana-cloud/metrics/prometheus/">Metrics — Prometheus</a> from the Grafana Cloud docs. To learn more about the <code>remote_write</code> parameter, please see <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write" target="_blank" rel="noopener noreferrer"><code>remote_write</code></a> from the Prometheus docs.</p> <h3 id="step-3-configuring-recording-rules">Step 3: Configuring recording rules</h3> <p>Using recording rules, you can precompute and cache frequently queried metrics. For example, if a dashboard panel uses a computationally intensive query like a <code>rate()</code>, you can create a recording rule that runs at a regular reduced interval and saves the result of the intensive query in a new time series. This avoids fetching and computing data every time the dashboard gets refreshed. To learn more about Prometheus recording rules, please see <a href="https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/" target="_blank" rel="noopener noreferrer">Recording Rules</a> from the Prometheus docs.</p> <p><strong>You should load the following recording rules before loading the dashboards in this guide.</strong> The dashboard queries use recording rules to reduce load on the Prometheus or Grafana Cloud Metrics servers, depending on where you&rsquo;re evaluating the rules.</p> <p>You can fetch the recording rule YAML file <a href="?tab=recording-rules">here</a>.</p> <h4 id="load-recording-rules-into-prometheus">Load recording rules into Prometheus</h4> <p>To load recording rules into Prometheus, add the following to your <code>prometheus.yml</code> configuration file:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">rule_files: - &#34;node_exporter_recording_rules.yml&#34;</code></pre> </div> </div> <p>Be sure to replace <code>node_exporter_recording_rules.yml</code> with the path to your Node Exporter recording rules YAML file.</p> <h4 id="load-recording-rules-into-grafana-cloud">Load recording rules into Grafana Cloud</h4> <p>To learn how to load recording rules into Grafana Cloud, please see <a href="/docs/grafana-cloud/alerts/alerts-rules/#prometheus-and-loki-rules-with-cortextool">Prometheus and Loki rules with cortextool</a>.</p> <h3 id="step-4-configuring-dashboards">Step 4: Configuring dashboards</h3> <p>This quickstart includes the following three dashboards:</p> <ul> <li>Nodes Dashboard</li> <li>USE Method / Node Dashboard</li> <li>USE Method / Cluster Dashboard</li> </ul> <p>To learn how to import these dashboards into Grafana, please see <a href="/docs/grafana/latest/dashboards/export-import/#importing-a-dashboard">Importing a dashboard</a> from the Grafana docs.</p> <p><strong>The dashboard queries depend on the recording rules defined in the previous step. Be sure to import these before importing the dashboards.</strong></p> <p>You can fetch the dashboards <a href="?tab=dashboards">here</a>.</p> <h3 id="step-5-configuring-alerts">Step 5: Configuring alerts</h3> <p>With Prometheus alerting rules, you can define alerts that fire when PromQL expressions breach some threshold or satisfy specified conditions over a period of time. For example, you can define a <code>HighRequestLatency</code> alert that fires when a request latency metric is greater than some threshold over a period of time. As soon as the alerting condition is triggered, the alert moves into <code>Pending</code> state. After satisfying the condition for the period of time defined by the <code>for</code> parameter, the alert moves into <code>Firing</code> state. You can configure routing and notifications for firing alerts using a tool like <a href="https://prometheus.io/docs/alerting/latest/alertmanager/" target="_blank" rel="noopener noreferrer">Alertmanager</a>. Alertmanager is also built-in to Grafana Cloud.</p> <p>You can fetch the alerting rule YAML file <a href="?tab=alerting-rules">here</a>.</p> <h4 id="load-alerting-rules-into-prometheus">Load alerting rules into Prometheus</h4> <p>To load alerting rules into Prometheus, add the following to your <code>prometheus.yml</code> configuration file:</p> <div class="code-snippet code-snippet__mini"><div class="lang-toolbar__mini"> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> </div><div class="code-snippet code-snippet__border"> <pre data-expanded="false"><code class="language-none">rule_files: - &#34;node_exporter_alerting_rules.yml&#34;</code></pre> </div> </div> <p>Be sure to replace <code>node_exporter_alerting_rules.yml</code> with the path to your Node Exporter alerting rules YAML file.</p> <h4 id="load-alerting-rules-into-grafana-cloud">Load alerting rules into Grafana Cloud</h4> <p>To learn how to load alerting rules into Grafana Cloud, please see <a href="/docs/grafana-cloud/alerts/alerts-rules/#prometheus-and-loki-rules-with-cortextool">Prometheus and Loki rules with cortextool</a>.</p> <h3 id="conclusion">Conclusion</h3> <p>In this quickstart you installed and ran Node Exporter on your Linux machine. You then configured Prometheus to scrape the system metrics exposed by Node Exporter. You loaded recording rules and alerting rules into Prometheus, and finally imported Grafana dashboards to visualize your Linux system metrics.</p> <p>If you&rsquo;re using Grafana Cloud, you can skip all of the steps in this guide by installing the Linux Server integration with the Grafana Agent. This integration embeds a preconfigured Node Exporter into the agent and automatically provisions Grafana dashboards and Prometheus alerting and recording rules, so you don&rsquo;t have to import them manually. To learn how to set up the Linux Server integration, please see <a href="/docs/grafana-cloud/quickstart/agent_linuxnode/">Monitoring a Linux host using the Linux host integration</a>.</p> <p>The dashboards, recording rules, and alerting rules were generated using the Node Exporter Mixin. Mixins are reusable templates for dasboards, recording rules, and alerts curated and designed by subject matter experts. To learn more, please see the <a href="https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin" target="_blank" rel="noopener noreferrer">Node Mixin</a> repository.</p>https://grafana.comMon, 01 Jan 0001 00:00:00 +0000https://grafana.com<p><strong>You should load the following recording rules before loading the dashboards in this guide.</strong> The dashboard queries use recording rules to reduce load on the Prometheus or Grafana Cloud Metrics servers, depending on where you&rsquo;re evaluating the rules.</p> <p>This quickstart includes the following recording rules:</p> <ul> <li><code>instance:node_num_cpu:sum</code></li> <li><code>instance:node_cpu_utilisation:rate1m</code></li> <li><code>instance:node_load1_per_cpu:ratio</code></li> <li><code>instance:node_memory_utilisation:ratio</code></li> <li><code>instance:node_vmstat_pgmajfault:rate1m</code></li> <li><code>instance_device:node_disk_io_time_seconds:rate1m</code></li> <li><code>instance_device:node_disk_io_time_weighted_seconds:rate1m</code></li> <li><code>instance:node_network_receive_bytes_excluding_lo:rate1m</code></li> <li><code>instance:node_network_transmit_bytes_excluding_lo:rate1m</code></li> <li><code>instance:node_network_receive_drop_excluding_lo:rate1m</code></li> <li><code>instance:node_network_transmit_drop_excluding_lo:rate1m</code></li> </ul> <a class="btn btn--primary br-5 mb-1" href="assets/node_rules.yaml" rel="noopener noreferrer">Download the following recording rules YAML file</a> <div class="code-snippet "><div class="lang-toolbar"> <span class="lang-toolbar__item lang-toolbar__item-active">YAML</span> <span class="code-clipboard"> <button x-data="app_code_snippet()" x-init="init()" @click="copy()"> <img class="code-clipboard__icon" src="/media/images/icons/icon-copy-small-2.svg" alt="Copy code to clipboard" width="14" height="13"> <span>Copy</span> </button> </span> <div class="lang-toolbar__border"></div> </div><div class="code-snippet "> <pre data-expanded="false"><code class="language-yaml">&#34;groups&#34;: - &#34;name&#34;: &#34;node-exporter.rules&#34; &#34;rules&#34;: - &#34;expr&#34;: | count without (cpu) ( count without (mode) ( node_cpu_seconds_total{job=&#34;node&#34;} ) ) &#34;record&#34;: &#34;instance:node_num_cpu:sum&#34; - &#34;expr&#34;: | 1 - avg without (cpu, mode) ( rate(node_cpu_seconds_total{job=&#34;node&#34;, mode=&#34;idle&#34;}[1m]) ) &#34;record&#34;: &#34;instance:node_cpu_utilisation:rate1m&#34; - &#34;expr&#34;: | ( node_load1{job=&#34;node&#34;} / instance:node_num_cpu:sum{job=&#34;node&#34;} ) &#34;record&#34;: &#34;instance:node_load1_per_cpu:ratio&#34; - &#34;expr&#34;: | 1 - ( node_memory_MemAvailable_bytes{job=&#34;node&#34;} / node_memory_MemTotal_bytes{job=&#34;node&#34;} ) &#34;record&#34;: &#34;instance:node_memory_utilisation:ratio&#34; - &#34;expr&#34;: | rate(node_vmstat_pgmajfault{job=&#34;node&#34;}[1m]) &#34;record&#34;: &#34;instance:node_vmstat_pgmajfault:rate1m&#34; - &#34;expr&#34;: | rate(node_disk_io_time_seconds_total{job=&#34;node&#34;, device!=&#34;&#34;}[1m]) &#34;record&#34;: &#34;instance_device:node_disk_io_time_seconds:rate1m&#34; - &#34;expr&#34;: | rate(node_disk_io_time_weighted_seconds_total{job=&#34;node&#34;, device!=&#34;&#34;}[1m]) &#34;record&#34;: &#34;instance_device:node_disk_io_time_weighted_seconds:rate1m&#34; - &#34;expr&#34;: | sum without (device) ( rate(node_network_receive_bytes_total{job=&#34;node&#34;, device!=&#34;lo&#34;}[1m]) ) &#34;record&#34;: &#34;instance:node_network_receive_bytes_excluding_lo:rate1m&#34; - &#34;expr&#34;: | sum without (device) ( rate(node_network_transmit_bytes_total{job=&#34;node&#34;, device!=&#34;lo&#34;}[1m]) ) &#34;record&#34;: &#34;instance:node_network_transmit_bytes_excluding_lo:rate1m&#34; - &#34;expr&#34;: | sum without (device) ( rate(node_network_receive_drop_total{job=&#34;node&#34;, device!=&#34;lo&#34;}[1m]) ) &#34;record&#34;: &#34;instance:node_network_receive_drop_excluding_lo:rate1m&#34; - &#34;expr&#34;: | sum without (device) ( rate(node_network_transmit_drop_total{job=&#34;node&#34;, device!=&#34;lo&#34;}[1m]) ) &#34;record&#34;: &#34;instance:node_network_transmit_drop_excluding_lo:rate1m&#34;</code></pre> </div> </div> <p>This recording rule YAML file was generated using the Node Exporter <a href="https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin" target="_blank" rel="noopener noreferrer">mixin</a>. It uses the <code>job=node</code> label selector to query metrics by default. If you need to use a different selector, modify the selector in <a href="https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/config.libsonnet" target="_blank" rel="noopener noreferrer">config.libsonnet</a> and regenerate the dashboard following the instructions in the mixin repository.</p>