A translation of the article was prepared ahead of the start of the course “Monitoring and Logging: Zabbix, Prometheus, ELK” .
Earlier we looked at a counter , a gauge, and a summary . Now let's talk about how the histogram works in Prometheus.The histogram has some similarities with the summary. A histogram is a combination of various counters. Like summary metrics, histogram metrics are used to track dimensional indicators of events, often their duration, using the method observe. Timekeeping tools are usually the same as bulletins. What they differ in is the processing of quantiles.The following is an example of a Prometheus tagged exposure formathandler :# HELP prometheus_http_request_duration_seconds Histogram of latencies for HTTP requests.
# TYPE prometheus_http_request_duration_seconds histogram
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.1"} 25547
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.2"} 26688
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.4"} 27760
prometheus_http_request_duration_seconds_bucket{handler="/",le="1"} 28641
prometheus_http_request_duration_seconds_bucket{handler="/",le="3"} 28782
prometheus_http_request_duration_seconds_bucket{handler="/",le="8"} 28844
prometheus_http_request_duration_seconds_bucket{handler="/",le="20"} 28855
prometheus_http_request_duration_seconds_bucket{handler="/",le="60"} 28860
prometheus_http_request_duration_seconds_bucket{handler="/",le="120"} 28860
prometheus_http_request_duration_seconds_bucket{handler="/",le="+Inf"} 28860
prometheus_http_request_duration_seconds_sum{handler="/"} 1863.80491025699
prometheus_http_request_duration_seconds_count{handler="/"} 28860
_sumand they _countwork in exactly the same way as for the summary, and can be used to get the average execution time over the past five minutes: rate(prometheus_http_request_duration_seconds_sum[5m]
/
rate(prometheus_http_request_duration_seconds_count[5m])
However, there are very rare cases where it is _sumabsent, for example, in some metrics from the MySQLd exporter.A notable part of the histogram is the time series _bucket, which are actually the histogram part of the metric. More specifically, these are counters that form a cumulative histogram. ledenote less than or equal to. Thus, 26688 requests took less than or equal to 200ms, 27760 requests took less than or equal to 400ms, and there were 28860 requests in total. The values in the bucket will be monotonically non-decreasing, and the +Infbucket will have the greatest value. +Infbucket must always be present and match the value _count.To calculate, say, a quantile of 0.9 (the 90th percentile), you should use:histogram_quantile(0.9,
rate(prometheus_http_request_duration_seconds_bucket[5m])
)
One big advantage of histograms over summaries is that you can aggregate buckets before calculating the quantile - taking care not to lose the label le:histogram_quantile(0.9,
sum without (handler)(
rate(prometheus_http_request_duration_seconds_bucket[5m])
)
)
In addition to aggregation, histograms are cheaper on the client, as the counters are quickly incremented. So why not use histograms always? There is a long answer, but the short version is that with the histograms you must first select your bucket, and the costs are transferred from the client to Prometheus itself due to the number of bucket elements. Ten buckets by default cover a typical web service with a delay ranging from milliseconds to a second, and in some cases you may need to change this. Here, for example, they were redefined to better track PromQL queries, which by default have a two-minute timeout. Having more than a dozen bucket-s will give more accurate results, but can also produce many time series. Especially in combination with other tags.When using a real-time monitoring system such as Prometheus, the goal should be to provide analytical value sufficient to make engineering decisions based on it. For example, to know that the delay of the 90th percentile has increased by 50 ms is more important than to know if it is now 562 ms or 563 ms, and usually ten bucket-s are enough for this. If you need an exact answer, you can always calculate it later from your logs. If there are too many bucket-s, they can be discarded during data processing, as was shown earlier . In more extreme cases, you can completely ignore the series of _buckets and rely on the average of _sumand _count.In conclusion, histograms allow the aggregate calculation of quantiles, although the number of elements must be taken into account. Remember that a summary without quantiles is a cheaper option if you do not need a histogram.
More about the course