Saya sarankan Anda membiasakan diri dengan decoding laporan oleh Roman Havronenko "ExtendedPromQL"
![](https://habrastorage.org/webt/ek/i2/uj/eki2uj2zonf5fkyegqdubqlvekc.png)
Tentang saya secara singkat. Nama saya Roman. Saya bekerja di CloudFlare, tinggal di London. Tapi saya juga pelindung VictoriaMetrics.
Dan saya penulis plugin ClickHouse untuk Grafana dan ClickHouse-proxy - ini adalah proxy kecil untuk ClickHouse.
![](https://habrastorage.org/webt/mo/n4/bd/mon4bdnqnfpdcrw5xdt9yxtfp5k.png)
, Β« Β» , β . , , . , β java - . , , .
![](https://habrastorage.org/webt/4_/t7/f8/4_t7f8-avlr_mqfrxlpxrzu3xo0.png)
. PromQL? PromQL β Prometheus Query Language. , Prometheus, time series , .
![](https://habrastorage.org/webt/sl/xm/xl/slxmxlenqvdk3zuwppxfws32w2s.png)
time series ? , .
:
![](https://habrastorage.org/webt/sm/cv/jt/smcvjtqzyel6bpo0nptwwjdfase.png)
( , ), .
. , . . . , 15 000 . time series .
![](https://habrastorage.org/webt/td/ui/bg/tduibg52iq_nhdsczrdsaz6wsrg.png)
"" () . , . , -, . . , , , Jay Silent Bob. , ; .
![](https://habrastorage.org/webt/5y/mx/ln/5ymxlnycfahn8v-25fykmpnpmk4.png)
. ClickHouse . , Β«StepsΒ», . . . , ; - -, , : Jay Silent Bob.
![](https://habrastorage.org/webt/4z/rb/rc/4zrbrcraa0o4n11b6nqe9wcnwg4.png)
, Grafana, , -, .
![](https://habrastorage.org/webt/jl/vy/ar/jlvyarnffgamdbzpw-jlsvghisi.png)
. . β . , time series ClickHouse, Grafana.
![](https://habrastorage.org/webt/ep/_j/sh/ep_jshq0tikq70umihqfx4h5f8w.png)
Graph Panel. Grafana, , .
![](https://habrastorage.org/webt/ce/xd/_p/cexd_pg4lgyoe6ewugorhtezalu.png)
β Grafana, ClickHouse, , . . . , . . , .
![](https://habrastorage.org/webt/4d/vj/km/4dvjkme86hiyazekbuuan1mdvq0.png)
. , ?
![](https://habrastorage.org/webt/o3/th/fp/o3thfpv96wg5r39fam8s2r1d-a0.png)
, .
![](https://habrastorage.org/webt/dn/62/af/dn62af7dmcccmjiiul3-_btzbcg.png)
, . , ? , , Grafana time series, - , time series β - labels.
![](https://habrastorage.org/webt/_r/zv/kc/_rzvkcgn6dgaurlq-e9ypktbrlc.png)
. Jay.
![](https://habrastorage.org/webt/nq/vf/kw/nqvfkwx1ho5vtvvfm8xad0jo31s.png)
. . .
![](https://habrastorage.org/webt/ax/ws/uz/axwsuz9m8lozzdd_gpaj2cq3mqu.png)
, , , , Prometheus PromQL. . . . Steps. Jay. , .
![](https://habrastorage.org/webt/0s/nn/qo/0snnqogfhgjcmxkagwzggfsf0o4.png)
Jay Silent Bob. ClickHouse runningDifference, . . , . .
![](https://habrastorage.org/webt/ba/__/ls/ba__ls6guurcuvg2apxapl0ceoq.png)
, . . 1,8 Silent Bob Jay.
![](https://habrastorage.org/webt/f7/rk/lk/f7rklkvtw5joxj7cheqrlnikxm8.png)
Prometheus , . , .
![](https://habrastorage.org/webt/26/x2/im/26x2immhsygbsrgdvbe4mrbhmui.png)
Grafana , PromQL. Rate Macros . Grafana Β«rateΒ», - . , - , , SQL- β . , .
![](https://habrastorage.org/webt/iv/ss/fl/ivssflyqwhf9nr9g4li56nsyxre.png)
, . ClickHouse, rate, time series: Silent Bob, Jay, time series. , .
![](https://habrastorage.org/webt/ez/de/_d/ezde_dum2gptsjbjct2cwe3iilw.png)
Prometheus sum (rate). ClickHouse , RateColumns, , Prometheus.
![](https://habrastorage.org/webt/ae/el/zp/aeelzpvytgwgqx53w-mvtvg4fh0.png)
PromQL , , , .
:
- Limited SELECT.
- JOIN.
- HAVING.
, , - PromQL, SQL , , , SQL. ? , .
![](https://habrastorage.org/webt/7u/9k/jq/7u9kjqnhre3bq9gv7nvfdbj5ozy.png)
. . , , , , .
![](https://habrastorage.org/webt/wm/mk/fy/wmmkfy7sxez3l9kqy8kbcxtwdjw.png)
β Extending PromQL.
![](https://habrastorage.org/webt/y7/gi/cn/y7gicneqli_dhkn-adm6wp0kseg.png)
VictoriaMetrics. VictoriaMetrics? time series , OpenSource, single cluster-. , , . . - 0,4 , Prometheus β 1,2-1,4.
Prometheus. InfluxDB, Graphite, OpenTSDB.
"", . . .
Prometheus Grafana, . . PromQL engine. Grafana Prometheus endpoint VictoriaMetrics , .
, VictoriaMetrics.
, .
![](https://habrastorage.org/webt/hf/qa/wo/hfqawo-ol7yyns3_ru3sngniqzu.png)
Omit interval param β Grafana. zoom-in/out , $__interval
. Grafana . VictoriaMetrics , . . .
![](https://habrastorage.org/webt/ys/j0/e1/ysj0e1hz9uzixz18hlzrsbl8moa.png)
β interval referencing. . , , , .
![](https://habrastorage.org/webt/gj/wc/tg/gjwctgfj1lfzzyzltg9fkzwdwxc.png)
rollup function. Rollup function time series time series. min, max avg. , , - outliers ( ) .
![](https://habrastorage.org/webt/ir/_u/mi/ir_umi41-5i66f_0qaihrohct6e.png)
irate rate, - , time series , . , , max avg.
![](https://habrastorage.org/webt/4w/ee/ty/4weety7gklowpxc-uaiv4ypzyxg.png)
default. Default β , Grafana, time series . ? - . , , . , success error. , error. default .
![](https://habrastorage.org/webt/ca/bs/su/cabssudrqxuinviag6b08ygllfs.png)
Keep_last_Value β , . Prometheus 5 , .
![](https://habrastorage.org/webt/xh/p1/vk/xhp1vkkknhw7imulfokfl4jdnqk.png)
Scrape_interval β , Prometheus , . , .
![](https://habrastorage.org/webt/qv/r_/ud/qvr_ud_7h-jx1jnkfuezncdptma.png)
Label replace β . , , . 5 , .
![](https://habrastorage.org/webt/nv/5x/rg/nv5xrgixmrzytgd0rysk7vvqll0.png)
, ? . . .
![](https://habrastorage.org/webt/w9/yp/jg/w9ypjg9rkgn-a-e_1cnodrn2bcc.png)
. , extended PromQL? Common Table Expressions. QR- (https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/ExtendedPromQL), , playground, VictoriaMetrics .
![](https://habrastorage.org/webt/jg/y5/he/jgy5hezln0mwu2rp3xqi9fu_gys.png)
? β . , . . - , , , JSON, find replace, . ? , , .
![](https://habrastorage.org/webt/nm/qj/ba/nmqjbaqwpsfevw27cb_vekd1wc8.png)
, Grafana , . Grafana?
![](https://habrastorage.org/webt/ep/zh/uw/epzhuw8yfbhi3adpubqaa2y9q7s.png)
: commonFilter , . , , Grafana . .
![](https://habrastorage.org/webt/kw/aj/6l/kwaj6lrv3en_ly9hhh2xrtzf3wk.png)
, . , , . https://github.com/grafana/grafana/pull/16694
![](https://habrastorage.org/webt/5c/y1/lt/5cy1ltsavquwd5g8e4i-su0flrk.png)
PromQL extended. , . ru (resource usage). , . . . . . , . , , , - .
![](https://habrastorage.org/webt/jz/ai/b-/jzaib-a71atxlrifwvr_azphzm8.png)
- . NodeExporter Grafana. , . . ., , , , , . ?
![](https://habrastorage.org/webt/jb/v6/md/jbv6mdppyv8l0_ztsvodfchh4iq.png)
, , . . , , Grafana.
![](https://habrastorage.org/webt/wt/4v/-a/wt4v-abwkorpzhyqkphqbdjqhhi.png)
, , ru, VictoriaMetrics. , CTE.
![](https://habrastorage.org/webt/bm/po/me/bmpomek_z-9_oa2yol1pp4w69kq.png)
, , . , , Grafana - . , , Grafana , - . - -. , - , . . .
, , , SRE. , , , , Grafana, . . , . 100 , - .
, Grafana, ? , . , , , , , ?
. . Grafana. , , NodeExporter, , . .
![](https://habrastorage.org/webt/nq/6d/ma/nq6dma-mxtherrcw5uz6da4mgrc.png)
. . . Grafana, JSON , . . . - , .
![](https://habrastorage.org/webt/uv/g5/80/uvg580w0egu0tyqliynpf7-6syi.png)
-, , Grafana . Grafana "", - - β . , .
![](https://habrastorage.org/webt/ed/46/1t/ed461t6tlckbtprrgwdis29tg6u.png)
VictoriaMetrics. . compression, time series , PromQL, , vertical scalability Thanos.
![](https://habrastorage.org/webt/hz/dx/iz/hzdxizdmbgw929apdc_nprroyh4.png)
:
. Grafana, 5 . . production. , , , , . . , , Grafana? , - β . , , . .
. . β , , , , , . - , , , . β , , , capacity planning - .
?
? , .
Grafana.
Grafana? Grafana DataSource.
- .
, Grafana . GET-, , , step. , , , . . , Grafana, .
! ! mapping , Grafana . , ?
.
, alert Grafana. alert . , , alerts Grafana?
Grafana - -, β , . Grafana , alertmanager.
Ya, saya menggunakannya, tetapi sepertinya lebih mudah diatur di Grafana, tetapi terima kasih atas sarannya!