👨‍👩‍👧 🤛🏿 ⏰ Mikhail Salosin. Golang Meetup. Using Go in the backend of the Watch + application 🙍🏽 🚴🏻 👨🏽‍⚕️

Mikhail Salosin (hereinafter - MS): - Hello everyone! My name is Michael. I work as a backend developer at MC2 Software, and I will talk about using Go in the backend of the Watch + mobile application.

Does anyone present like hockey?

Then this app is for you. It is for Android and iOS, used to watch broadcasts of various sporting events online and in recordings. Also in the application there are various statistics, text broadcasts, tables for conferences, tournaments and other information useful for fans.

Also in the application there is such a thing as video moments, that is, you can see the acute moments of matches (goals, fights, shootouts, etc.). If you don’t want to watch the whole broadcast, you can watch only the most interesting.

What was used in development?

The main body was written in Go. The API that the mobile clients communicated with was written in Go. Also, a service was written on Go for sending push notifications to mobile. We also had to write our own ORM, which we may someday talk about. Well, some small services are written on Go: resizing and image uploading for the editors ...

We used Postgres (PostgreSQL) as a database. The interface for editors was written in Ruby on Rails using the ActiveAdmin gem. The import of statistics from the statistics provider is also written on Ruby.

For system API tests, we used the Python unittest (Python). Memcached is used to throttle API payment requests, Chef to control configuration, Zabbix to collect and monitor internal system statistics. Graylog2 - for collecting logs, Slate is an API documentation for clients.

Protocol selection

The first problem we encountered: we had to choose a protocol for the interaction of the backend with mobile clients, based on the following points ...

The most important requirement: data on clients must be updated in real time. That is, everyone who is currently watching the broadcast should receive updates almost instantly.
For simplicity, we accepted that data that is synchronized with clients is not deleted, but hidden using special flags.
All sorts of rare requests (like statistics, lineups, team statistics) are received by ordinary GET requests.
Plus, the system was supposed to calmly withstand 100 thousand users at the same time.

Based on this, we had two protocol options:

Websockets. But we did not need channels from the client to the server. We only needed to send updates from the server to the client, so a web socket is a redundant option.
Server-Sent Events (SSE) came up just right! It is quite simple and basically satisfies everything that we need.

Server-Sent Events

A few words about how this thing

works ... It works on top of the http connection. The client sends a request, the server responds with Content-Type: text / event-stream and does not close the connection with the client, but continues to write data to the connection:

Data can be sent in a format agreed with the clients. In our case, we sent in this form: in the event field the name of the changed structure (person, player) was sent, and in the data field - JSON with new, changed fields for the player.

Now about how the interaction itself works.

First of all, the client determines when the last synchronization was made with the service: it looks at its local database and determines the date of the last change recorded from it.
He sends a request with this date.
In response, we send him all the updates that have occurred on this date.
After that, he makes a connection to the live channel and does not close until he needs these updates:

We send him a list of changes: if someone scored a goal - we change the score of the match, got injured - also sent in real time. Thus, in the event stream of the match, customers instantly receive relevant data. Periodically, so that the client understands that the server has not died, that nothing has happened to it, we send a timestamp every 15 seconds - so that it knows that everything is in order and there is no need to reconnect.

How is the live connection served?

First of all, we create a channel into which updates with a buffer will come.
After that, we subscribe this channel to receive updates.
Set the correct header so that the client knows that everything is ok.
ping. timestamp .
, . timestamp, , .

The first problem we encountered was the following: for each connection opened with the client, we created a timer that ticked once every 15 seconds - it turns out if we had 6 thousand connections with one machine (with one API server), 6 thousand timers were created. This led to the fact that the machine did not hold the necessary load. The problem was not so obvious to us, but they helped us a little, and we eliminated it.

As a result, now we have ping coming from the same channel from which update comes.

Accordingly, there is only one timer that ticks once every 15 seconds.

Here are some helper functions - sending the header, ping, and the structure itself. That is, the name of the table (person, match, season) and the information about this record are transmitted here:

Mechanism for sending updates

Now a little about where the changes come from. We have several people, editors, who watch the broadcast in real time. They create all the events: someone was removed, someone was injured, some kind of replacement ...

With the help of CMS, the data gets into the database. After that, the database using the Listen / Notify mechanism notifies the API servers about this. API servers are already sending this information to clients. Thus, in fact, we have only a few servers connected to the database and there is no special load on the database, because the client does not interact directly with the database in any way:

PostgreSQL: Listen / Notify

The Listen / Notify mechanism in Postgres allows you to notify subscribers of events that some event has changed - some kind of record has been created in the database. To do this, we wrote a simple trigger and function:

When inserting or changing a record, we call the notify function on the data_updates channel, transfer the table name and the identifier of the record that was changed or inserted there.

For all tables that should be synchronized with the client, we define a trigger that, after changing / updating the record, calls the function indicated on the slide below.
How does the API subscribe to these changes?

The Fanout mechanism is created - it sends messages to the client. He collects all the customer channels and sends out updates that he received through these channels:

Here the standard pq library, which connects to the database and says that it wants to listen to the channel (data_updates), checks that the connection is open and everything is fine. I omit error checking to save space (do not check fraught).

Next, we set the Ticker asynchronously, which will send ping every 15 seconds, and begin to listen to the channel to which we subscribed. If we got a ping, we publish this ping. If we received a record, then we publish this record to all subscribers of this Fanout.

How does fan-out work?

In Russian, this translates as a “splitter”. We have one object that registers subscribers who want to receive any updates. And as soon as an update to this object arrives, it spreads this update to all subscribers it has. Simple enough:

How it is implemented on Go:

There is a structure, it is synchronized using Mutex's. It has a field that saves the status of the Fanout connection to the database, that is, at the moment it listens and will receive updates, as well as a list of all available channels - map, the key of which is the channel and struct in the form of values (in fact, it not used in any way).

Two methods - Connected and Disconnected - allow you to tell Fanout that we have a connection to the base, it appeared and that the connection to the base is disconnected. In the second case, you need to disconnect all clients and tell them that they can no longer listen to anything and that they reconnect, since the connection with them has closed.

There is also a Subscribe method that adds a channel to listeners:

There is an Unsubscribe method that removes a channel from listeners if the client disconnects, as well as a Publish method that allows you to send a message to all subscribers.

Question: - What is transmitted through this channel?

MS: - A model is transmitted that has changed or ping (essentially just a number, integer).

MS:- You can send anything, publish any structure, it just turns into JSON and that’s it.

MS: - We get a notification from Postgres - it contains the table name and identifier. By the name of the table we get and the identifier we get the record we need, and already this structure is sent for publication.

Infrastructure

What does it look like in terms of infrastructure? We have 7 iron servers: one of them is completely dedicated to the base, virtual computers are spinning on the remaining six. There are 6 copies of the API: each virtual machine with the API runs on a separate iron server - this is for reliability.

We have two frontends on which Keepalived is installed to improve accessibility, so that in case of which one frontend can replace the other. Another two copies of the CMS.

There is also an importer of statistics. There is a DB Slave from which backups are periodically made. There is Pigeon Pusher - the application that sends pushies to customers, as well as infrastructure things: Zabbix, Graylog2 and Chef.

In fact, this infrastructure is redundant, because 100 thousand can be served with fewer servers. But there was iron - we used it (we were told that it is possible - why not).

Pros of Go

After we worked on this application, such obvious advantages of Go were revealed.

Cool http library. Using it, you can create quite a lot already out of the box.
Plus, the channels that allowed us to very easily implement the mechanism for sending notifications to customers.
The wonderful Race detector allowed us to eliminate several critical bugs (staging-infrastructure). Everything that works on staging is running, compiled with the Race key; and, accordingly, we can see what potential problems we have on the staging infrastructure.
Minimalism and simplicity of language.

We are looking for developers! If someone wants - please.

Questions

Question from the audience (hereinafter - B): - It seems to me that you missed one important point regarding the Fan-out. I understand correctly that when you send a response to a client, you are blocked if the client does not want to read?

MS: - No, we are not blocking. Firstly, we have it all behind nginx, that is, there are no problems with slow clients. Secondly, the client has a channel with a buffer - in fact, we can put up to a hundred updates there ... If we cannot write to the channel, then it deletes it. If we see that the channel is blocked, then we just close the channel, and that’s it - the client will reconnect if there is any problem. Therefore, in principle, blocking does not occur here.

Q: - Could you immediately send to Listen / Notify a record, not an identifier table?

MS:- Listen / Notify has a limit of 8 thousand bytes per preload, which it sends. In principle, it would be possible to send if we were dealing with a small amount of data, but it seems to me that the way [as we do] is simply more reliable. Limitations are in Postgres itself.

Q: - Do customers receive updates on matches that they are not interested in?

MS:- In general, yes. As a rule, there are 2-3 matches in parallel, and then quite rarely. If the client is watching something, then usually he is watching the match that is going on. Then, on the client there is a local database into which all these updates add up, and even without an Internet connection, the client can see all the past matches for which he has updates. In fact, we synchronize our database on the server with the local database of the client so that it can work offline.

Q: - Why did you make your ORM?

Alexey (one of the developers of “Watch +”):- At that time (it was a year ago) ORM was less than now, when there are quite a lot of them. Of the majority of existing ORMs, what I dislike the most is that most of them work on empty interfaces. That is, the methods that in these ORMs are ready to take on anything: structure, structure pointer, number, something irrelevant at all ...

Our ORM generates structures based on the data model. Itself. And therefore, all methods are concrete, do not use reflection, etc. They accept structures and expect to use the structures that come.

Q: - How many people participated?

MS: - At the initial stage, two people participated. Somewhere in June we started, in August the main part was ready (first version). In September there was a release.

AT:- Where you describe SSE, you do not use timeout. Why is that?

MS: - To be honest, SSE is still an html5 protocol: the SSE standard is designed to communicate with browsers, as I understand it. It has additional features so that browsers can reconnect (and so on), but we do not need them, because we had clients that could implement any logic for connecting and receiving information. We did more likely not SSE, but something similar to SSE. This is not the protocol itself.
There was no need. As far as I understand, clients implemented the connection mechanism from scratch. In principle, they didn’t care.

Q: - What additional utilities did you use?

MS:- The most active we used govet and golint, so that the style was unified, as well as gofmt. They didn’t use anything else.

Q: - With what did you debug?

MS: - By and large, debugging was done using tests. No debugger, GOP we did not use.

Q: - Can you return the slide where the Publish function is implemented? Single-letter variable names do not bother you?

MS: - No. They have a fairly "narrow" scope. They are not used anywhere else (except here) (except for the insides of this class), and it is very compact - it takes only 7 lines.

Q: - Somehow it’s still not intuitive ...

MS:- No, no, this is a real code! It's not about style. It’s just such a utilitarian, very small class - there are only 3 fields inside the class ...

MS: - By and large, all the data that is synchronized with customers (seasonal matches, players) does not change. Roughly speaking, if we are going to do another kind of sport in which it will be necessary to change the match, we will just consider everything in the new version of the client, and the old versions of the client will be banned.

Q: - Are there any third-party packages for dependency management?

MS: - We used go dep.

Q: - There was something about the video in the subject of the report, but no about the video in the report.

MS: - No, I have nothing in the topic about the video. It is called “Look +” - this is the name of the application.

AT:- You said that you are streaming to customers? ..

MS: - We did not do streaming video. This was completely done by Megaphone. Yes, I did not say that the application is megaphone.

MS: - Go - to send all the data - by score, by match events, statistics ... Go - this is the entire backend for the application. The client must find out from somewhere which link to use for the player so that the user can watch the match. We have links to videos and streams that are prepared.

A bit of advertising :)

Thank you for staying with us. Do you like our articles? Want to see more interesting materials? Support us by placing an order or recommending to your friends cloud-based VPS for developers from $ 4.99 , a unique analog of entry-level servers that was invented by us for you: The whole truth about VPS (KVM) E5-2697 v3 (6 Cores) 10GB DDR4 480GB SSD 1Gbps from $ 19 or how to divide the server? (options are available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).

Dell R730xd 2 times cheaper at the Equinix Tier IV data center in Amsterdam? Only we have 2 x Intel TetraDeca-Core Xeon 2x E5-2697v3 2.6GHz 14C 64GB DDR4 4x960GB SSD 1Gbps 100 TV from $ 199 in the Netherlands!Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - from $ 99! Read about How to Build Infrastructure Bldg. class c using Dell R730xd E5-2650 v4 servers costing 9,000 euros for a penny?

Mikhail Salosin. Golang Meetup. Using Go in the backend of the Watch + application