मैं अलेक्जेंडर वालेलकिन द्वारा 2019 के अंत की रिपोर्ट के डिकोडिंग से परिचित होने का प्रस्ताव रखता हूं "विक्टोरियामैट्रिक्स में अनुकूलन"
विक्टोरियामीटर एक टाइम सीरीज़ के रूप में डेटा को संग्रहीत करने और संसाधित करने के लिए एक तेज़ और स्केलेबल डेटाबेस मैनेजमेंट सिस्टम है (इस समय के रिकॉर्ड रिकॉर्ड समय और मूल्यों का एक सेट, उदाहरण के लिए, सेंसर की स्थिति या मैट्रिक्स के संग्रह के आवधिक मतदान) के माध्यम से प्राप्त किया जाता है।
स्लाइड्स
. – . GitHub . Go . . fast, quick .
VictoriaMetrics. ? .
:
- , VictoriaMetrics.
- , .
- , .
- : .
- , VictoriaMetrics. bitset- Go.
VictoriaMetrics - ? , . . , – . ClickHouse, ClickHouse. , , : MergeTree, , .
VictoriaMetrics .
— . . , . VictoriaMetrics .
VictoriaMetrics — . . VictoriaMetrics, .
, VictoriaMetrics , . Go, .
, ? . — (timestamp, ), . – float64.
. ? -.
. : __name__="cpu_usage" – , instance="my-server" — , , datacenter="us-east" — , .
, -. (timestamp, value). t1, t3, t3, ..., tN — timestamps, 10, 20, 12, ..., 15 — . cpu-usage .
? - ?
- DevOps CPU, RAM, , rps, . .
- IoT – , geo coordinates -.
- – .
- , . , VictoriaMetrics , .
- . , ; ; , ; . .
- . , . .
- Healthcare – , ..
, , , , , , . .
? ?
, . . (timestamp, value) . API , -, , regexp. , , .
, SQL . , SQL , . , PromQL, InfluxQL, Flux, Q. , - . PromQL, , . Prometheus.
VictoriaMetrics.
. . .
, , label=value . .
- TSDB, . timeseries_ids , label=value. , timeseries_ids.
, select-.
timeseries_ids , label=value, .- data points
timeseries_ids. - - data points, . .
. timeseries_ids . VictoriaMetrics, , :)
. , . , ? , . , .
. , . ? label=value, label value — . – timeseries_ids, label=value.
timeseries_ids, label=value.
timeseries_ids label=value, label=regexp. ? timeseries_ids label=value.
. . .
getMetricIDs . label=value. metricIDs.
? , invertedIndex. (map), slice int-. label=value.
: metricIDs label=value, label=value, metricIDs . intersectInts, . .
, . . ? , . . . .
. . , , – out of memory error. .
, LevelDB, RocksDB.
, , .
LevelDB RocksDB – Google Facebook. LevelDB. Facebook LevelDB , RocksDB. RocksDB Facebook , RocksDB MySQL. MyRocks.
LevelDB. ? label=value. – , label=value.
c label=value, timeseries_ids. timeseries_ids, label=prefix, range scan, . . . , label=prefix timeseries_ids.
, Go. . LevelDB.
, . . , map . label=value. label=value metricIDs . .
, . VictoriaMetrics LevelDB. .
? LevelDB , . slice metricIDs. — slice .
LevelDB GetValues , label=value. timeseries_ids. timeseries_ids slice timeseries_ids. , , map' .
, LevelDB C. C- Go . . , , go, 1-5 , . VictoriaMetrics – :)
. mergeset.
Mergeset MergeTree. ClickHouse. , mergeset timeseries_ids . Mergeset Go. VictoriaMetrics GitHub. mergeset /lib/mergeset. , .
API mergeset LevelDB RocksDB. . . .
mergeset . , VictoriaMetrics production .
?
– high churn rate. – . , . .
– . , , . , , , . 4 . , , 100 400 . .
, . , , . 40 , , , 40 .
. , idle, . user space, kernel space . , . 7-8 .
40 8 = 320 . 100, 32 000 400.
Kubernetes. , Kubernetes . Kubernetes . . deployment . . , high-cardinality. VictoriaMetrics .
high churn rate. - high churn rate production? .
, Kubernetes, deployment, . . . Kubernetes - id’ deployment' label.
? , deployment' , deployment_id. .
, , , , , . – high churn rate.
high churn rate , - . .
? . . . . - , . . , .
, . , , id .
, , . . . , , .
. id , .
, , . .
, . VictoriaMetrics . . , . - , partition , , . . . VictoriaMetrics , . , .
. , timeseries_ids , , .
? – . . . label=value, . timeseries_ids .
. N , , , . id .
10 . , label=value N . , labels , Kubernetes.
– . . key=value . . . , . , . . . , ClickHouse. .
– timeseries_ids. , . timeseries_ids label=value.
– . . intersectInts slices — a b. slices.
. slice a, slice b. . , , . result.
? — . , slice a b , . , .
map. map. map slice a. slice b. – slice b map. , .
? , . . . slices. slice 2 , , .
, , map.
– overhead . . , VictoriaMetrics intersection map. , map map.
? Go . . . hash , HashMap. hash’ . VictoriaMetrics.
bitset, . slices. bitset. slice. slice. . . . . , map add has.
, , map, - , , 10 , map VictoriaMetrics.
, map. .
, , .
, – , . . . , ids VictoriaMetrics. , . , , out of memory error .
. , VictoriaMetrics, lib/uint64set. VictoriaMetrics, timeseries_id 64- , 32 32 .
, .
API. . API VictoriaMetrics. . . . , VictoriaMetrics.
add, . has, . del, . len, . clone . appendto slice timeseries_ids.
. set :
ItemsCount – , set. , , VictoriaMetrics bitset.
– buckets. slice bucket32. hi . 32 . slice — b16his buckets bucket16 .
16 64- . bitsets 16 .
Bucket64 uint64. . bucket16 2^16=65536 . 8, 8 . 8, 1000 uint64 . . . Bucket16 – 8- .
, .
uint64 . 32 , 32 . buckets. 32 bucket . , add b32 buckets. 32 . true, , . false, . .
bucket hi-, addAlloc, bucket, bucket-.
b32.add. . 16 , 16 .
16 . . add, bucket16.
, . uint64 id slice bit, bitmask. 64- , , . , , . , ids 10 maps.
VictoriaMetrics . , production.
– , , , , . . production, . , VictoriaMetrics , .
bitset. C++ vector bool, bitset. ?
, . bitset ids timeseries, VictoriaMetrics. , 32 . 32 . , . . C++ , , . , , .
. - ClickHouse . , C++ - . , , , , 32 .
. InfluxDB?
. , InfluxDB 10 high cardinality , , , . , VictoriaMetrics 1 GB , InfluxDB 10 Gb. .
, InfluxDB – Flux InfluxQL. PromQL, VictoriaMetrics. PromQL – Prometheus.
– , InfluxDB , fields . . . .
VictoriaMetrics . -. – – (timestamp, value), – label=value. fields measurements. , , , , , InfluxDB, , . , , .
. , - , , , ?
Go' map'. , , , . . , .
! ! . Wildberries. . . , , , , , , , . . , - ? 2 — bitset , , , ? , ?
. . , . , ?
. , . , .
. , , . churn rate , . . , .
– , – . Prometheus . Prometheus . , - label timeseries. , , , , VictoriaMetrics. VictoriaMetrics , - , .
, , . timeseries-, . . – intersect . , , .
?
.
LevelDB mergeset . . ? – . , . – 5 . 100 000 000 , , .
, . . , , . ? - VictoriaMetrics, ClickHouse . ., ?
, . , ? , , , – . . . . .
, . . , , . , , - ClickHouse , , , . , - , , , ? , , .
, . , Prometheus recording rules. . . , , . VictoriaMetrics , Prometheus, recoding rules.
, . , Go, . . . , . , - . , , . , ClickHouse timeseries-, .
. , , - Graphite Carbon. , . . , . . , , , , , . Prometheus, VictoriaMetrics . ? , ?
. . , (downsampling). . -, downsampling - : - , - , , . , . , . .
– , VictoriaMetrics, ClickHouse, , , , . VictoriaMetrics – 50 000 000 . . . . 20 , , . VictoriaMetrics ClickHouse downsamling.
, VictoriaMetrics . production 0,4 0,8 . – timestamp + . , .
. . ?
. . – . Graphite, , . OpenTSDB . InfluxDB – . VictoriaMetrics – , Prometheus . VictoriaMetrics remote storage Prometheus . .
, , , — , , . DevOps- , 30 , , , . high frequency trading systems, .
VictoriaMetrics DevOps-, , . , – high frequency trading systems.
! . PromQL?
. VictoriaMetrics PromQL. PromQL, MetricsQL. YouTube. Monitoring Meetup .
Telegram VictoriaMetrics.