System Design Primer (Teil 1 mit Microservice-Ergänzungen)

Vor ein paar Jahren habe ich diesen Primer übersetzt, um einen ihrer Artikel zu schließen. De facto wurde diese Option zu einer offiziellen Übersetzung. Die Entwicklung dieser Übersetzung wird jedoch durch die Tatsache behindert, dass sie in Google Drive geschrieben und zur Bearbeitung geschlossen wurde. Heute habe ich mir die Zeit genommen, diesen gesamten Text mit Hilfe des Habr-Editors in das MD-Format umzuformatieren. Ich freue mich, ihn hier zu veröffentlichen und werde ihn bald an aktuelle Mitwirkende weitergeben.

Wörterbuch

Node — —
Content — — e
Traffic — — /,
Hardware — —
Instance — — . API
Headers — — ( TCP , HTTP )

(Performance) vs (Scalability)

. , , .

• , .
• , , .

• A word on scalability
• Scalability, availability, stability, patterns

(Latency) vs (Throughput)

- .

, .

, .

• Understanding latency vs throughput

(Availability) vs () Consistency

CAP

, :

• — , .
• — , . , .
• — , , (node) .

, . .

CP — (consistency and partition tolerance) — +
, , timeout ( ). . . , -, .

AP — (availability and partition tolerance) — +
, . ( ), .

, . , .

• CAP theorem revisited
• A plain english introduction to CAP theorem
• CAP FAQ
• Eventual consistency

, . . CAP — .

, , . best effort (A best effort approach is taken)

memcached. real-time, VoIP, . , , , .

, , , ( ). .

DNS Email. .

, . , . .

RDBMSes. , .

• Transactions across data centers

: .

(Fail-over)

- (Active-passive)
(heartbeats) . , IP .

(downtime) , . (hot) (cold) .

— shoot the other node in the head (STONITH), . , , .

- master-slave — -.

-
- , , .

, DNS IP . , .

-, — master-master — -.

• .
• , .

Master-slave master-master
.

(Domain name system)

DNS — IP . www.example.com 127.0.0.1.

DNS . , DNS . DNS , , - . DNS .

• NS- (name server) — DNS /.
• MX- (mail exchange) — .
• A- (address) — IP .
• CNAME- (canonical) — (example.com www.example.com) A-.

CloudFlare Route 53 DNS . DNS , :

• 
  — ,
  —
  — A/B

DNS

• DNS , , .
• DNS , DNS, .
• DNS .

• DNS architecture
• Wikipedia
• DNS articles

(Content delivery network)

Content delivery network (CDN) — , . , CDN HTML, CSS, JS, .

CDN :

• CDN

Push CDN

Push CDNs . , CDN () CDN . .

Push CDN . CDN, .

Pull CDN

Pull CDN . , . CDN- .

TTL — time-to-live, . Pull CDN , CDN, , CDN , .

Pull CDN , .

: CDN.

• CDN , .
• .

:

• Globally distributed content delivery
• The differences between push and pull CDNs
• Wikipedia

(Load balancer)

(, ), , .

:

• ;
• ;
• ;

() HAProxy.

:

• SSL — .
  — .
  — X.509
• — . , , .

, - - .

:

• /
• Round robin or weighted round robin
• 4 OSI
• 7 OSI

Layer 4

, . IP , . Network Address Translation (NAT).

Layer 7

, , .

.

4- , , . , .

(Horizontal scaling)

, . ( ). - .

:

• ,
• (SQL, NoSQL) (Redis, Memcached)

:

• , .
• , .
• , .

• NGINX architecture
• HAProxy architecture guide
• Scalability
• Wikipedia
• Layer 4 load balancing
• Layer 7 load balancing
• ELB listener config

(-) — Reverse proxy (web server)

. , .

:

• — , IP , .
• — , .
• SSL — .
• —
• —
• —
  — HTML/CSS/JS
  —
  —
  —

vs

• . , .
• .
• NGINX HAProxy layer 7 .

:

• .
• . .

• Reverse proxy vs load balancer
• NGINX architecture
• HAProxy architecture guide
• Wikipedia

/ (Application layer)

. API, .

.

— Microservices

, , . API . Pinterest, , : , , , , .

«Microservice Architecture» . , : -, , (endpoints) .

«» — . , , . , , . , . , , , .

, — , , , HTTP. - . . .

, , (monolithic style): , . Enterprise : ( HTML javascript-), ( , ) . HTTP , , , HTML , . .

— . , , namespace-. . .

, , , . , , . , , . , .

: . , . .

, . , , Unix. , .

, , , , . , , , . , .

, , . , .

, , . : — , .

, — . , , — , - remote procedure call (RPS).

— . , , , . , , - , . , - , , , , — , .

— . Published Interface. . .

, . , , API (coarser-grained), . , - , .

, . , . , , .

, , UI , . , - - . , , . (Conway's Law) .

« , - ( ) , »
— Melvyn Conway, 1967

(Conway's Law)

. , -, , . - , : user-experience, project management.

,

, — www.comparethemarket.com. - , .

, . , . , . , - . , . .

?

«» , , «». , , . , « » (, ), .. 12 (. .: , ). , .
, , . , (1 12 1 1 ) , . (. .: , ; , , « », ).

,

, , , , . .

, , . , " , ", . , , , , .. .

. — , . , — -.

, , .

(Smart endpoints and dumb pipes)

, . Enterprise Service Bus (ESB). ESB- , , -.

: . , , (decoupled) (cohesive), : Unix- — , . , WS-* BPEL, REST- .

— HTTP API . Ian Robinson: «Be of the web, not behind the web».

, , , (, , Unix). IT-.

— . — RabbitMQ ZeroMQ . — , .

. . «» , . .

. , — . . , .

, , . Node.js ? . C++ real-time ? . , ? .

, -, . .

, , . , , -, , . , -. , git github - , - .

Netflix — , . , , , . , , .

, . Tolerant Reader Consumer-Driven Contracts , . Consumer-Driven . , . : , — YAGNI.

— , . , , 24/7. , , . Netflix — , . 3 — , .

SOA

, , Service Oriented Architecture (SOA), . , .. , SOA. , , , SOA , , , «SOA» , , - ESB, .

, SOA ( ESB, , ), .

, , , . Tolerant Reader — . — — , .

SOA , «SOA», SOA, , , SOA. , , SOA , , .

,

JVM — . , , . , « » .

, DSL- . , .

. , . , enterprise-: «» . «» . , .

, , . Bounded Context Domain-Driven Design (DDD). DDD . , , , .

, . , . , , . : , . Polyglot Persistence. Polyglot Persistence , .

, . , . .

, (temporal coupling), , , . , , , (eventual consistency) .

— , . , . , , .

, , vs

, , , . HTTP ATOM.

, . , IETF, , open-source .

, , .

. AWS , .

, , Continuous Delivery Continuous Integration. , , . .

Continuous Delivery, . , , . , (automated deployment).

, () . , Continuous Delivery — , .

, , — . , , , , .

(operations folk). , , . , , — open source Netflix, , Dropwizard, .

(Design for failure)

, . - . , . , .. . , , user experience. Simian Army Netflix () .

, . , , , .

, , , . , (, ), - (, ). , .

, .. . .

, . , . , , , , , . , , .

, , . , (/) -: , ..

, (downtime). . : , . , www.guardian.co.uk — . Netflix API .

, , () . : , , .

, , . - , , «» ? — , . : , , .

- Guardian — , , . , , API . , . — . . , .

— , , . , , . , , . , , .

(granular) . . (redeploy) , . . , , , . , , . , , .

?

, . , — , enterprise . , .

— Amazon, Netflix, The Guardian, the UK Government Digital Service, realestate.com.au, Forward comparethemarket.com. 2013 , , , , Travis CI. , , , , . ( SOA, , , SOA , , .)

, , — . , , , .

, . , , «» . , , .. . , , , , .

, - . , . . . , , , , . , . . .

, , . , , .

. , , , . , , , . , .

, , , . , . ( , .. .)

, . , . , , , , , .

— Service Discovery

Consul, Etcd, Zookeeper, (, ). Health checks ( ), , HTTP .

: application layer

• .
• .

• Intro to architecting systems for scale
• Crack the system design interview
• Service oriented architecture
• Introduction to Zookeeper
• Here's what you need to know about building microservices

(RDBMS)

SQL
ACID —

• Atomicity — .

  
  

• Consistency — .

  
  

• Isolation — .

  
  

• Durability — , .

  
  

. - , - , , , , SQL .

- — Master-slave replication
, , . , . , .

: master-slave:

• .

- — Master-master replication
. , .

: master-master replication

• , , .
• master-master ( ACID), - .
• , .
• . : , master-slave, master-master.

:

• .
• . , .
• , .
• .

• Scalability, availability, stability, patterns
• Multi-master replication

— Federation

. , . : , , . , . . , .

:

• .
• .
• .

Scaling up to your first 10 million users

// — Sharding

. . , , . .

: sharding

• SQL .
• . , . .
• .

• The coming of the shard
• Shard database architecture
• Consistent hashing

. , . (PostgreSQL, Oracle)
, . .

(1000:1), .

:

• , .
• . .

Denormalization

SQL

SQL .
.

• Benchmark — .
• Profile — , .

• MySQL .
• CHAR VARCHAR .
• CHAR , VARCHAR, , .
• TEXT . TEXT . TEXT .
• INT 2^32 4 .
• DECIMAL .
• BLOBS, .
• NOT NULL .

Use good indices

• (SELECT, GROUP BY, ORDER BY, JOIN) .
• B- , , .
• .
• , .
• , .

• 
  — .
• 
  — .

• Tips for optimizing MySQL queries
• Is there a good reason i see VARCHAR(255) used so often?
• How do null values affect performance?
  - Slow query log

NoSQL

NoSQL - , , . , . NoSQL ACID.

BASE NoSQL. CAP BASE , .

• Basically available — .
• Soft state — .
• Eventual consistency — .

, SQL NoSQL, , NoSQL . .

- — Key-value store

: hash table

Key-value store O(1),. , . - .

, in-memory . , , .

- .

• Key-value database
• Disadvantages of key-value stores
• Redis architecture
• Memcached architecture

/ — Document store

: key-value store

XML, JSON, binary . . API .

, , , . , .

MongoDB CouchDB SQL . DynamoDB - .

.

• Document-oriented database
• MongoDB architecture
• CouchDB architecture
• Elasticsearch architecture

— Wide column store

: map ColumnFamily<RowKey, Columns<ColKey, Value, Timestamp>>

— . ( SQL).

.

• SQL & NoSQL, a brief history
• Bigtable architecture
• HBase architecture
• Cassandra architecture

— Graph database

: graph

, — . , .

, .

• Graph database
• Neo4j
• FlockDB