Decryption of the 2016 report by Vitaliy Levchenko "Prometheus monitoring of microservice applications"
Prometheus, unlike classical systems, makes it easy to raise and maintain monitoring of rapidly changing and complexly organized systems. I will talk about implementation experience, pitfalls and unexpected behavior, show how to quickly configure the entire system, including notifications and dashboards.
, . , , , job' — . : — , , .. , , , , .
Prometheus Google Borgmon , . , , — . — , . — .
, Prometheus .
, , , . , . Prometheus , . :
. . . .
, , developer. . . .
, , , .
? . . docker. , , , .
, . . , . , , – . ? , – . , production .
– production . production 10 . , continuous delivery. stages. . . .
, . . .
, . , , .
. , – . , - environments – . stage environment, environment, production.
, , . . , 20 , .
. The twelve factor app – Production. , , . – .
, .
. . . , , , . – . . . . , . . , . . .
– . . , , , Graphite, StatsD. , , push based, . .
– , , . . , , .
, , . , . Graphite, . . , 100 000 – . – , 100 000 . – . , 21 . , , , .
production Zabbix - ? . – . . . Zabbix – , . . , .
, . , , Zabbix . Nagios . , , . . . , , - .
, , , . Zabbix . , , Redis. .
, , . InfluxDB . . Graphite, InfluxDB , , .
InfluxDB , . . production. . . , . . .
, . , , . . . (?) . . .
, , , . . . , . . , – . .
Riemann . : « CPU ». CPU – 100 %, 0 %. .
– , CollectD . , . InfluxDB. - , - , - , CollectD , . , , .
InfluxDB , time-series , .
Prometheus. ?
Google, , , Google Borgmom, , Facebook. , , , . . .
, . . production ( , , , ) , . . Prometheus .
. . Prometheus , . .
, . deprecate- . Grafana. Grafana , .
, Prometheus . . . . , 20 000 – . 24- 400 000 . 3 , 1 200 000. .
production . - . , . . , - .
, , , . . . environment. server, . handler . . . Prometheus . , .
, , . , , , . . , . .
? . : « environment , ». .
. - , , . , , , – .
Prometheus , . . , .
. . , 4 . . , . Zookeeper, , … 95 % , 20 100 %. . – . . . , .
- pull metrics push metrics. , pull metrics .
StatsD – Graphite, . StatsD , . . .
, exporter . , , 50 000 . . , , production, .
StatsD exporter Prometheus . StatsD, StatsD, gauge etc, .
push gateway push’. Prometheus. StatsD, push gateway .
. , Prometheus, , . . . , , . . Amazon, Kubernetes, Mesos, Consul . . . , .
, Ansible, . . . - , , , . , Prometheus. reload , .
, . , , . javascript . , . .
. - .
Prometheus . . open source. , Postgres exporter – , . Node exporter, .
, . , . , . .
systemd. .
, . , .
. , . . , , . , , . Prometheus . . . Influx, .
. .
. , - , . , , . – –rm – r <storage path/*. . . , , production. , , .
federation, , federation , .
. . . Prometheus . .
openTSDB . , . openTSDB , , , , Hadoop.
, , , , histogram summary. , .
, , 100 – . 100 300 – . , 300 – . histogram. , … , , , . . . , histogram , . 10 , 10 . - . , recording rules. , , . , .
Summary – . Summary . Summary , . , , , . . , . , 99- , , . . , . .
summary . . . , .
-. . .
? ? , , , :
- . , , . . , latency , , . 99 % . , , - . . .
- .
- .
- .
- + .
?
- . , . . 10 , 10 , 10 , . ( Service Discovery)
- , . Grafana . , . : « ». 10 . , . . , , .
- – , . . . . , , , , .
- – , , , , .
- , Prometheus . . , Prometheus docker. . . – . . .
! ! Zabbix. , . , . , , Zabbix. environment, ?
. – environment, . ( Kubernetes Service Discovery)
. ., , developer, ? . . environment?
.
environment?
environment. environment. environment. , environment, stable – environment. ( Kubernetes Service Discovery)
, . . . - , , . , - . . . , - , , . , ?
Zabbix production, , . , , , -. full time , . Zabbix, , , 3-4- . 3-4- , Zabbix – . .
, 3 Zabbix?
.
. . developers Prometheus?
.
? ?
.
. environment, - sender, Prometheus .
sender. Prometheus . API http’, . Consul, Prometheus , . .
. , , - Prometheus?
, . .
. docker’, , , , - , . overhead, . . . , ?
?
.
health handler, . , , . , , .
! ! 4 . StatsD, , , . StatsD, Influx’ Telegraf?
, StatsD, Influx.
Ok. , Grafana . , . Go, . , .
open source, .
GitHub , . . federation, ?
, . .
Ok. -?
, .
, . ? ?
. , . 100 000 . , 200 000-300 000. , , , , . Zabbix, . Influx , . . , queries .
?
?
, - , , .
, - . , . . .