# Tutorials # Alerting based on metrics In this tutorial we will create alerts on the `ping_request_count` metric that we instrumented earlier in the [Instrumenting HTTP server written in Go](./instrumenting_http_server_in_go.md) tutorial. For the sake of this tutorial we will alert when the `ping_request_count` metric is greater than 5. Check out real world [best practices](../practices/alerting.md) to learn more about alerting principles. Download the latest release of Alertmanager for your operating system from [here](https://github.com/prometheus/alertmanager/releases). Alertmanager supports various receivers like `email`, `webhook`, `pagerduty`, `slack` etc through which it can notify when an alert is firing. You can find the list of receivers and how to configure them [here](/docs/alerting/latest/configuration/). We will use `webhook` as a receiver for this tutorial, head over to [webhook.site](https://webhook.site) and copy the webhook URL which we will use later to configure the Alertmanager. First let's setup Alertmanager with the webhook receiver. > alertmanager.yml ```yaml global: resolve_timeout: 5m route: receiver: webhook_receiver receivers: - name: webhook_receiver webhook_configs: - url: '' send_resolved: false ``` Replace `` with the webhook that we copied earlier in the alertmanager.yml file and run the Alertmanager using the following command. `alertmanager --config.file=alertmanager.yml` Once the Alertmanager is up and running navigate to [http://localhost:9093](http://localhost:9093) and you should be able to access it. Now that we have configured the Alertmanager with webhook receiver let's add the rules to the Prometheus config. > prometheus.yml ```yaml global: scrape_interval: 15s evaluation_interval: 10s rule_files: - rules.yml alerting: alertmanagers: - static_configs: - targets: - localhost:9093 scrape_configs: - job_name: prometheus static_configs: - targets: ["localhost:9090"] - job_name: simple_server static_configs: - targets: ["localhost:8090"] ``` Note that the `evaluation_interval`,`rule_files` and `alerting` sections were added to the Prometheus config. `evaluation_interval` defines the intervals at which the rules are evaluated, `rule_files` accepts an array of yaml files that defines the rules and the `alerting` section defines the Alertmanager configuration. As mentioned in the beginning of this tutorial we will create a basic rule where we want to raise an alert when the `ping_request_count` value is greater than 5. > rules.yml ```yaml groups: - name: Count greater than 5 rules: - alert: CountGreaterThan5 expr: ping_request_count > 5 for: 10s ``` Now let's run Prometheus using the following command. `prometheus --config.file=./prometheus.yml` Open [http://localhost:9090/rules](http://localhost:9090/rules) in your browser to see the rules. Next run the instrumented ping server and visit the [http://localhost:8090/ping](http://localhost:8090/ping) endpoint and refresh the page at least 6 times. You can check the ping count by navigating to the [http://localhost:8090/metrics](http://localhost:8090/metrics) endpoint. To see the status of the alert visit [http://localhost:9090/alerts](http://localhost:9090/alerts). Once the condition `ping_request_count > 5` is true for more than 10s the `state` will become `FIRING`. Now if you navigate back to your `webhook.site` URL you will see the alert message. Similarly Alertmanager can be configured with other receivers to notify when an alert is firing. # Instrumenting HTTP server written in Go In this tutorial we will create a simple Go HTTP server and instrument it by adding a counter metric to keep count of the total number of requests processed by the server. Here we have a simple HTTP server with `/ping` endpoint which returns `pong` as response. ```go package main import ( "fmt" "net/http" ) func ping(w http.ResponseWriter, req *http.Request) { fmt.Fprintf(w,"pong") } func main() { http.HandleFunc("/ping", ping) http.ListenAndServe(":8090", nil) } ``` Compile and run the server ```bash go build server.go ./server ``` Now open `http://localhost:8090/ping` in your browser and you must see `pong`. [![Server](/assets/docs/tutorial/server.png)](/assets/docs/tutorial/server.png) Now let's add a metric to the server which will instrument the number of requests made to the ping endpoint. The counter metric type is suitable for this as we know the request count doesn’t go down and only increases. Create a Prometheus counter ```go type metrics struct { pingCounter prometheus.Counter } func newMetrics(reg prometheus.Registerer) *metrics { m := &metrics{ pingCounter: promauto.With(reg).NewCounter( prometheus.CounterOpts { Name: "ping_request_count", Help: "No of requests handled by Ping handler", }), } return m } ``` Next, let's update the ping Handler to increase the count of the counter using `metrics.pingCounter.Inc()`. ```go func ping(m *metrics) func(w http.ResponseWriter, req *http.Request) { return func(w http.ResponseWriter, req *http.Request) { m.pingCounter.Inc() fmt.Fprintf(w, "pong") } } ``` Then register the metrics (in this case, only one counter) with a Prometheus registry and expose the metrics. ```go func main() { reg := prometheus.NewRegistry() m := newMetrics(reg) http.HandleFunc("/ping", ping(m)) http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{})) http.ListenAndServe(":8090", nil) } ``` The `prometheus.MustRegister` function registers the pingCounter with the default registry. To expose the metrics, the Go Prometheus client library provides the promhttp package. `promhttp.Handler()` provides an `http.Handler` which exposes the metrics registered in the default registry. The sample code is now: ```go package main import ( "fmt" "net/http" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promauto" "github.com/prometheus/client_golang/prometheus/promhttp" ) type metrics struct { pingCounter prometheus.Counter } func newMetrics(reg prometheus.Registerer) *metrics { m := &metrics{ pingCounter: promauto.With(reg).NewCounter( prometheus.CounterOpts{ Name: "ping_request_count", Help: "No of request handled by Ping handler", }), } return m } func ping(m *metrics) func(w http.ResponseWriter, req *http.Request) { return func(w http.ResponseWriter, req *http.Request) { m.pingCounter.Inc() fmt.Fprintf(w, "pong") } } func main() { reg := prometheus.NewRegistry() m := newMetrics(reg) http.HandleFunc("/ping", ping(m)) http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{})) http.ListenAndServe(":8090", nil) } ``` Run the example ```bash go mod init prom_example go mod tidy go run server.go ``` Now hit the localhost:8090/ping endpoint a couple of times and then send a request to localhost:8090/metrics to see the metrics. [![Ping Metric](/assets/docs/tutorial/ping_metric.png)](/assets/docs/tutorial/ping_metric.png) Here, the `ping_request_count` shows that the `/ping` endpoint was called 3 times. The default registry comes with a collector for Go runtime metrics, and that is why we see other metrics like `go_threads`, `go_goroutines`, etc. We have built our first metric exporter. Let’s update our Prometheus config to scrape the metrics from our server. ```yaml global: scrape_interval: 15s scrape_configs: - job_name: prometheus static_configs: - targets: ["localhost:9090"] - job_name: simple_server static_configs: - targets: ["localhost:8090"] ``` prometheus --config.file=prometheus.yml # Understanding metric types Prometheus supports four types of metrics: * Counter * Gauge * Histogram * Summary ## Counter Counter is a metric value that can only increase or reset i.e. the value cannot reduce than the previous value. It can be used for metrics like the number of requests, no of errors, etc. Type the below query in the query bar and click execute. go\_gc\_duration\_seconds\_count [![Counter](/assets/docs/tutorial/counter_example.png)](/assets/docs/tutorial/counter_example.png) The rate() function in PromQL takes the history of metrics over a time frame and calculates how fast the value is increasing per second. Rate is applicable on counter values only. rate(go\_gc\_duration\_seconds\_count[5m]) [![Rate Counter](/assets/docs/tutorial/rate_example.png)](/assets/docs/tutorial/rate_example.png) ## Gauge Gauge is a number which can either go up or down. It can be used for metrics like the number of pods in a cluster, the number of events in a queue, etc. go\_memstats\_heap\_alloc\_bytes [![Gauge](/assets/docs/tutorial/gauge_example.png)](/assets/docs/tutorial/gauge_example.png) PromQL functions like `max_over_time`, `min_over_time` and `avg_over_time` can be used on gauge metrics ## Histogram Histogram is a more complex metric type when compared to the previous two. Histogram can be used for any calculated value which is counted based on bucket values. Bucket boundaries can be configured by the developer. A common example would be the time it takes to reply to a request, called latency. Example: Let's assume we want to observe the time taken to process API requests. Instead of storing the request time for each request, histograms allow us to store them in buckets. We define buckets for time taken, for example `lower or equal 0.3`, `le 0.5`, `le 0.7`, `le 1`, and `le 1.2`. So these are our buckets and once the time taken for a request is calculated it is added to the count of all the buckets whose bucket boundaries are higher than the measured value. Let's say Request 1 for endpoint “/ping” takes 0.25 s. The count values for the buckets will be. > /ping | Bucket | Count | | --------- | ----- | | 0 - 0.3 | 1 | | 0 - 0.5 | 1 | | 0 - 0.7 | 1 | | 0 - 1 | 1 | | 0 - 1.2 | 1 | | 0 - +Inf | 1 | Note: +Inf bucket is added by default. (Since the histogram is a cumulative frequency 1 is added to all the buckets that are greater than the value) Request 2 for endpoint “/ping” takes 0.4s The count values for the buckets will be this. > /ping | Bucket | Count | | --------- | ----- | | 0 - 0.3 | 1 | | 0 - 0.5 | 2 | | 0 - 0.7 | 2 | | 0 - 1 | 2 | | 0 - 1.2 | 2 | | 0 - +Inf | 2 | Since 0.4 is below 0.5, all buckets up to that boundary increase their counts. Let's explore a histogram metric from the Prometheus UI and apply a few functions. prometheus\_http\_request\_duration\_seconds\_bucket{handler="/graph"} [![Histogram](/assets/docs/tutorial/histogram_example.png)](/assets/docs/tutorial/histogram_example.png) `histogram_quantile()` function can be used to calculate quantiles from a histogram histogram\_quantile(0.9,prometheus\_http\_request\_duration\_seconds\_bucket{handler="/graph"}) [![Histogram Quantile](/assets/docs/tutorial/histogram_quantile_example.png)](/assets/docs/tutorial/histogram_quantile_example.png) The graph shows that the 90th percentile is 0.09, To find the histogram_quantile over the last 5m you can use the rate() and time frame histogram_quantile(0.9, rate(prometheus\_http\_request\_duration\_seconds\_bucket{handler="/graph"}[5m])) [![Histogram Quantile Rate](/assets/docs/tutorial/histogram_rate_example.png)](/assets/docs/tutorial/histogram_rate_example.png) ## Summary Summaries also measure events and are an alternative to histograms. They are cheaper but lose more data. They are calculated on the application level hence aggregation of metrics from multiple instances of the same process is not possible. They are used when the buckets of a metric are not known beforehand, but it is highly recommended to use histograms over summaries whenever possible. In this tutorial, we covered the types of metrics in detail and a few PromQL operations like rate, histogram_quantile, etc. # Visualizing metrics using Grafana In this tutorial we will create a simple dashboard using [Grafana](https://github.com/grafana/grafana) to visualize the `ping_request_count` metric that we instrumented in the [previous tutorial](./instrumenting_http_server_in_go.md). If you are wondering why one should use a tool like Grafana when one can query and see the graphs using Prometheus, the answer is that the graph that we see when we run queries on Prometheus is to run ad-hoc queries. Grafana and [Console Templates](https://prometheus.io/docs/visualization/consoles/) are two recommended ways of creating graphs. ## Installing and Setting up Grafana. Install and Run Grafana by following the steps from [here](https://grafana.com/docs/grafana/latest/installation/requirements/#supported-operating-systems) for your operating system. Once Grafana is installed and run, navigate to [http://localhost:3000](http://localhost:3000) in your browser. Use the default credentials, username as `admin` and password as `admin` to log in and setup new credentials. ## Adding Prometheus as a Data Source in Grafana. Let's add a datasource to Grafana by clicking on the gear icon in the side bar and select `Data Sources` > ⚙ > Data Sources In the Data Sources screen you can see that Grafana supports multiple data sources like Graphite, PostgreSQL etc. Select Prometheus to set it up. Enter the URL as [http://localhost:9090](http://localhost:9090) under the HTTP section and click on `Save and Test`. ## Creating our first dashboard. Now we have successfully added Prometheus as a data source, Next we will create our first dashboard for the `ping_request_count` metric that we instrumented in the previous tutorial. 1. Click on the `+` icon in the side bar and select `Dashboard`. 2. In the next screen, Click on the `Add new panel` button. 3. In the `Query` tab type the PromQL query, in this case just type `ping_request_count`. 4. Access the `ping` endpoint few times and refresh the Graph to verify if it is working as expected. 4. In the right hand section under `Panel Options` set the `Title` as `Ping Request Count`. 5. Click on the Save Icon in the right corner to Save the dashboard.