We make extensive use of microservice architecture, although we do not consider it a panacea, and a little more than 2 years ago we began to switch to the Go language. It is relatively simple and, in my opinion, very well suited for creating simple, small and fast microservices. This simplicity has a downside: because of it, there are many ways to solve the same problem.

It would seem, how much can one microservice that goes to the database differ from another microservice that goes to the neighboring database? For example, one team uses Go 1.9, glide, the standard database / sql and one project structure, while another team uses Go 1.13, modules, sqlx and, of course, another project structure.

When one microservice in a company is different from another, and it, in turn, is different from a third, this slows down development. And slow development is a ~~loss~~ for optimization.

My name is Alexey Partilov, I am a technical leader of a web development team at Lamoda. In this article I will tell you how we deal with the variability of about 40 of our microservices on Go. This article will be useful for developers who just join Go and don’t know where to start a more complex project than helloworld.

Initially, Lamoda worked on a single boxed monolith in PHP. Then part of the functions began to move to the new Python and PHP services. Now we have monoliths in PHP with a large and complex business logic. These applications are engaged in the automation of our operational activities: delivery, photo studio, affiliate service. We also have a monolith in Java, which automates many complex processes in the warehouse.

, , .

e-commerce Python Golang. , UI Python+Django.

, . .

, . backend- , , , , .

0. Spec first

, : , OpenAPI.

Backend-, , - . .
Frontend- backend, . mock backend .
boilerplate- c , . , , , .

OpenAPI- go-swagger ( gogi). opensource, , .

, . , , , .

, docker-. , . docker. Makefile:

.PHONY: generate
generate:
    #    
    rm -rf internal/generated/*
    docker run -it \
    #     
    -v $(PWD)/internal/generated:$(DOCKER_SOURCE_DIR)/internal/generated \
    #    OpenAPI 
    -v $(PWD)/specs:$(DOCKER_SOURCE_DIR)/specs \
    #    gogi
    -v $(PWD)/gogi.yaml:$(DOCKER_SOURCE_DIR)/gogi.yaml \
    #  workdir
    -w="$(DOCKER_SOURCE_DIR)" \
    gotools.docker.lamoda.ru/gogi:v1.3.2 generate

1.

. , , . , -. — , :

/
├── cmd
├── deployments
├── internal
├── migrations
├── specs
├── tests
├── go.mod
├── go.sum
├── Dockerfile
├── main.go
...

, ( Postgres), production-, Kafka API. ? . .

cookiecutter. . . , .

2. Go modules

1.11 Go go modules. 1.14 production-ready. glide, go modules, , , Go-. Go modules $GOPATH/pkg/mod GOPATH.

Go (build, get, test). , build . .

golang , , . , .

1. go modules. Go 1.13, $GOPATH, go mod

go env -w GOMODULE111="on"

2. RSA- . RSA- c , , ssh-agent. .

ssh-add <  RSA >

3. Go proxy. Go 1.13, - proxy.golang.org, . , .

go env -w GOPRIVATE=<  >

4. go.mod-. go.mod . (, glide), mod- lock-. go.mod — , .

go mod init <  >

, , , github.com/spf13/cobra.

5. . . , , , . , .

go test ./...

3. Docker-

Lamoda Kubernetes-, Docker- . , Go- Docker- golang. c Docker-. Docker- , .

ARG version
FROM golang:$version

ENV GOOS linux
ENV GOARCH amd64
ENV CGO_ENABLED 0
ENV GO111MODULE on

#      
RUN go get github.com/axw/gocov/gocov && \
   go get github.com/AlekSi/gocov-xml@d2f6da892a0d5e0b587526abf51349ad654ade51 && \
   go get golang.org/x/tools/cmd/goimports && \
   curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s v1.23.7 && \
   go get -u github.com/jstemmer/go-junit-report@af01ea7f8024089b458d804d5cdf190f962a9a0c && \
   rm -rf /go/pkg/mod/

#            
COPY ./ssh/id_rsa /root/.ssh/id_rsa
COPY ./ssh/id_rsa.pub /root/.ssh/id_rsa.pub
# Copy Lamoda Root certificate needed to go to the corporate sites without
# SSL warning
COPY ./certs/LamodaCA.crt /usr/local/share/ca-certificates/LamodaCA.crt

RUN echo "StrictHostKeyChecking no" > /root/.ssh/config && \
   chmod 600 /root/.ssh/id_rsa && \
   chmod 600 /root/.ssh/id_rsa.pub && \
   chmod 755 /root/.ssh && \
   update-ca-certificates && \
   #   insteadOf.       Bitbucket  .
   git config --global url.ssh://git@stash.lamoda.ru:7999.insteadOf https://stash.lamoda.ru/scm

CMD ["/bin/sh"]

golang- . golang production-. , ( ) : debian alpine, go, . Docker- , Docker multistage.

, Docker multistage — Docker, , , , ..

FROM gotools.docker.lamoda.ru/base-mod:1.14.0 as build

ENV GOOS linux
ENV GOARCH amd64
ENV CGO_ENABLED 0
ENV GO111MODULE on

WORKDIR /go/src/stash.lamoda.ru/ecom/discounts.endpoint
#     
COPY go.mod .
COPY go.sum .

#  go-. -    Docker          go.mod  go.sum.
RUN go mod download
COPY . .

RUN make build

#   docker        ca-certificates.crt
FROM scratch
COPY --from=build /go/src/stash.lamoda.ru/ecom/discounts.endpoint/discounts /bin/discounts
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENTRYPOINT ["/bin/discounts"]

4.

, , Postgres. , . , DBA, . . , . SQL- DBA DevOps production .

- Jira, migrate. , . , , migrate Docker- — .

— migrations <№ >_<>.[up|down].sql

migrations
├── 00001_init.down.sql
├── 00001_init.up.sql
├── 00002_create_ui_users_table.down.sql
├── 00002_create_ui_users_table.up.sql
...

up — , .. ,
down — , .. .

SQL-, .

«»:

BEGIN;
CREATE SEQUENCE ui_users_id_seq INCREMENT BY 1 MINVALUE 1 START 1;

CREATE TABLE ui_users (
  id INT NOT NULL,
  username VARCHAR(180) NOT NULL,
  username_canonical VARCHAR(180) NOT NULL,
  email VARCHAR(180) NOT NULL,
  email_canonical VARCHAR(180) NOT NULL,
  enabled BOOLEAN NOT NULL,
  salt VARCHAR(255),
  password VARCHAR(255) NOT NULL,
  last_login TIMESTAMP(0) WITHOUT TIME ZONE DEFAULT NULL,
  confirmation_token VARCHAR(180) DEFAULT NULL,
  password_requested_at TIMESTAMP(0) WITHOUT TIME ZONE DEFAULT NULL,
  roles TEXT NOT NULL,
  PRIMARY KEY(id)
);
COMMIT;

«»:

BEGIN;
DROP SEQUENCE public.ui_users_id_seq;

DROP TABLE ui_users;
COMMIT;

, :

migrate -database ${DB_DSN} -path db/migrations up

migrate create -ext sql -dir migrations -seq create_table_foo

, , . : . , , . migrate .

5. development-

Lamoda , , . / , . , Postgres . Postgres . / development , Docker Compose.

Compose development . , :

Postgres,
migrate, Postgres,
dev- .

. , . , , .

Docker-compose :

version: "3.7"
services:
  #   dev- 
  gift-certificates-dev:
    container_name: gift-certificates-dev
    #       ,    pull  docker-
    build:
      context: ../
      #    Dockerfile  multistage,       
      #  .          ,
      #             
      #    go run.
      target: build
    #     environment  .
    env_file:
      - local.env
    #   
    depends_on:
      - gift-certificates-db
    #       .     
    #   go run
    volumes:
      - "..:/app/"
    working_dir: "/app"
    #    
    command: "go run main.go"
    depends_on:
      - gift-certificates-db
  #    Postgres
  gift-certificates-db:
    container_name: gift-certificates-db
    image: postgres:11.4
    #      
    #      env_file,    default.env
    #   ,       
    environment:
      - POSTGRES_DB=gift_certificates
      - POSTGRES_USER=gift_certificates
      - POSTGRES_PASSWORD=gift_certificates
      - POSTGRES_PORT=5432
    #     docker 
    #  , ,          docker 
    ports:
      - 6543:5432
  #     
  gift-certificates-migrate:
    container_name: gift-certificates-migrate
    image: "migrate/migrate:v4.4.0"
    depends_on:
      - gift-certificates-db
    volumes:
      - "../migrations:/migrations"
    command: ["-path", "/migrations/", "-database", "$SERVICE_DB_DSN", "up"]

Docker-compose Makefile :

#   
#          
.PHONY: test
test: dev-migrate
    go test -cover -coverprofile=coverage.out ./...
    go tool cover -html=coverage.out -o coverage.html

#   dev 
.PHONY: dev-server
dev-server:
    docker-compose -f deployments/docker-compose.yaml up -d gift-certificates-dev

#       
.PHONY: dev-migrate
dev-migrate:
    docker-compose -f deployments/docker-compose.yaml run --rm --service-ports gift-certificates-migrate

#   dev 
.PHONY: dev-down
dev-down:
    docker-compose -f deployments/docker-compose.yaml down

6.

Go gofmt, , : goimports, go-critic, gocyclo .. , IDE CI . golangci-lint, Go . , -.

, «». , , «» , .

curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s -- -b $(go env GOPATH)/bin v1.23.7

run:
  tests: false
  #         
  skip-dirs:
    - generated
  skip-files:
    - ".*easyjson\\.go$"
output:
  print-issued-lines: false

issues:
  #      
  new-from-rev: 7cdb5ce7d7ebb62a256cebf5460356c296dceb3d
  exclude-rules:
    - path: internal/common/writer.go
      linters:
        - structcheck
      #       
      text: "`size` is unused"

golangci-lint run ./...

golangci-lint pre-commit, .

7. Pre-commit hook’

Pull Request : , / , . bash- git pre-commit . , -. : .

python pre-commit:

pip install pre-commit

, .

-. go:

repos:
- repo: https://github.com/golangci/golangci-lint
  rev: v1.23.7
  hooks:
  - id: golangci-lint

git pre-commit hook':

pre-commit install

golangci-lint . hook' . all-files , staging'e:

pre-commit run --all-files

hook' test runner'. https://pre-commit.com

8.

Gonkey — , Lamoda. , . , Gonkey, habr.ru “Gonkey — ”.

Gonkey :

HTTP- , ,
, ( YAML-),
( , Gonkey ),
Allure-.

GitHub.

9.

, «» . : , , , .

, Confluence . , « ». cookiecutter.

, cookiecutter — python-, jinja- . jinja- .

cookiecutter https://stash.lamoda.ru/gotools/cookiecutter-go

, , , .. . cookiecutter , .

Lamoda . . , , .

— . , ( ), ( ).

, , , , .

, / , , .

Unify it: how Lamoda makes its Go services consistent

0. Spec first

1.

2. Go modules

3. Docker-

4.

5. development-

6.

7. Pre-commit hook’

8.

9.

More articles: