Good afternoon, dear reader! This article arose from an attempt to look at operational monitoring from analytics. It will be useful to you if:

something from the following list is familiar: Grafana, InfluxDB, Prometheus, Zabbix, or another monitoring system with similar ideas;
You are not an expert / professional / confident middle in various types of time series display and statistics (otherwise, it will be very boring for you);
there is a desire to look at some aspects “under the microscope”.

If you decide that the conditions above are met, welcome to cat.

“Tell me, please, where should I go from here?”
“Where do you want to go?” - answered the Cat.
“I don't care ...” said Alice.
“Then it doesn't matter where to go,” said the Cat.
“... just to get somewhere,” Alice explained.
“You’re sure to go somewhere,” said the Cat. “You just have to go long enough.”
Alice in Wonderland by Lewis Carroll

The journey begins ...

. , , , , . , «?» . ? , ( , )! , .

, RFC1065:

…
3.2.3.3. Counter

This application-wide type represents a non-negative integer which

Rose & McCloghrie [Page 8]

RFC 1065 SMI August 1988

monotonically increases until it reaches a maximum value, when it
wraps around and starts increasing again from zero. This memo
specifies a maximum value of 2^32-1 (4294967295 decimal) for
counters.

3.2.3.4. Gauge

This application-wide type represents a non-negative integer, which
may increase or decrease, but which latches at a maximum value. This
memo specifies a maximum value of 2^32-1 (4294967295 decimal) for
gauges.
…

, , 1988- , : . - , , , Prometheus.

. — : , , . — - . - …

UI Grafana, , , - , $__interval, , , . — , 4k. , , — . , , .

, ( ) « » . :

, “Step count” 200 300 (, , . : 10 , :

	“”	$group_time
InfluxQL	WHERE time>now()-...	GROUP BY time(...)
	1h	20s	2
	1d (24h)	5m	30
	1w	1h	360
	1month	3h	1080

, ! ( ) , , .

. ( Influx’, ):

InfluxDB, , : An instantaneous rate of change is equivalent to a derivative ( ), . , , , , (average rate of change). . , , mean ( ), , . , «» , - . , , , , , .

«» : , last, ( InfluxQL, promql ). last . , . , , 00 . non_negative_derivative(first(...)) . , , — , — : ( ) .

, ? .

mean, InfluxDB : max, min, last, , , — stddev. prometheus min_over_time, max_over_time, stddev_over_time, Influx ( , ). Last, , , , , mean: - , mean , «».

Obviously, max and min will work so-so independently. But what if you combine these two and add the average to them? It turns out that this idea is not something new, it is generally ancient, like pyramids. Of the “classic” examples in monitoring systems, I recall trends in Zabbix (I intentionally looked for a link to an older version) and Graphite Whisper Rollup , as well as the earlier RRDtool , which was also used in Cacti , where for some reason min was missing by default. Perhaps, despite all the following text, this is one of the best approaches. It is quite informative, not too noisy, intuitive, cheap in implementation, relatively universal and available in almost all modern systems.

Examples of similar rendering:

in zabbix
Grafana

, min mean ( max avg) , , Grafana Legend -> Values , , , .

«»

, , stddev — standard deviation, - . , , . , , «» . , (, , , , ).

, : error bar. — “ ”, 1-, 2- 3- (25% , 75% ). , ? :) , , Grafana. , pandas seaborn ( plotly).

, , « » , , . “the Datasaurus Dozen animation” Google.

, , , . , , , .

, « » . ? seaborn! , : violinplot. , « » « -» , , - , : violinplot Grafana.

: , , ? , ? ? , ? - .

, , . : Google SRE book. « » (, , ): latency, traffic, errors, saturation. , - ( ), . : saturation % 100, : 100, «» , 0 — , , , . , , — , .

, : . , , , /. (, , ): pull push. , . , , , push — Riemann-tools, pull — Prometheus.

Telegraf, (input-plugins) , (output-plugins) . , discovery-plugins— input-plugins, . , , LLD (low-level discovery) Zabbix, . «»: ( ). (, prometheus-operator), , . , , … Kubernetes, — Influxdata. - ( API/RPC) snap, , , ( , ).

, pull- -, . , . , , , . . , : (, Riemann ). . , , , , (, - ). Graphite / statsd, , , UI. Prometheus recording rules Thanos Compact, TICK- , , : Continues Queries InfluxDB TICKscript Kapacitor. , , — , Histogram Summary Prometheus, Accumulator interface Telegraf.

B , , , , :