🖕 🧘 🕵🏽 Offer in London in one day: how to get it and what to do after moving 🧙🏻 🕵🏿 👩🏾‍💼

Hello, Habr!

We have big plans for 2020. We intend to actively develop Badoo and Bumble, so we are seriously expanding the technical team. And today we announce the massive hiring of PHP developers in our London office.

In 2017, we tried a new search format - hiring event: we bring cool developers to Moscow, conduct interviews in one day and immediately make an offer to suitable candidates. All expenses for a trip to the capital, of course, are incurred.

The format has worked well, and we have many open positions again, so we are announcing a new PHP Hiring Event.

The rules are the same: show a high test result before March 1, successfully pass the interview on March 21 or 22 in Moscow - and on the same day receive an offer to the Badoo London office.

UPD: the test is completed, thanks to all participants! We have already notified the finalists and warned that the dates of the interview are postponed indefinitely for the comfort and safety of candidates.

Under the cut, I’ll tell you:

more about the test;
what projects we are engaged in: photo optimization, video streaming, machine learning for letters, transition to new versions of PHP and much more.

If you are a PHP shnik and want to move to London, welcome to kat!

How to pass the test

The test consists of five tasks. Exactly 90 minutes are allotted to the decision: it will not work to delay or pause the process. Before you begin, make sure you have enough time.

In three tasks, you need to write code / SQL queries, and in the remaining two we expect to see detailed answers to questions. Test published on HackerRank. We recommend practicing on the test tasks of the platform in order to get comfortable with the interface.

Decisions are made until 2020-03-01 23:59:59 UTC. Based on the test results, we will select approximately 60 candidates whom we will invite to Moscow for an interview.

The company covers all expenses for a trip to Moscow, as well as expenses associated with moving to London. We help get working visas to the family members of our new colleague, pay for the flight, rent a house for the duration of the search, pay a cash bonus for the move and help in improving English. You can get acquainted with the team already at the interview stage: the developers themselves conduct interviews at the hiring event.

What the server development team does

We are responsible for the backend of dating services Badoo and Bumble. In addition to swipe left and right to find a partner, the applications have video streaming, chat with audio messages, searching for people nearby, integration with payment services and much more. A relatively simple interface hides a lot of logic in PHP and Go (and it gets more and more).

Our team has more than 40 people. The work can be divided into two large parts:

new functionality in applications,
technical projects - improvement of tools, optimization, work with technical debt.

I will talk about the most interesting projects that we have implemented over the past two years.

Cool chips in the product

The market for dating applications is dynamic and competitive, therefore our main goal is the development of the product: it must remain interesting and safe for users.

Learned to hide too "personal" photos

Some users send chat photos of an erotic nature, even if the interlocutor did not ask for such gifts.

The task that we had to solve was to teach the application to distinguish “private” photos (18+) from ordinary ones and show them chatted in the chat, allowing the recipient to decide for himself whether he wants to see them in their entirety.

Together with the R&D team, we created a service on Go with a neural network that can recognize "private" photos. If the likelihood that the photo is erotic is higher than the set threshold, a special flag is put in the message and the client handles this situation accordingly: shows a blurred stub asking if the recipient wants to see what is inside.

We run all tasks through A / B tests, and also carefully cover the key points of the new functionality with statistics. So, we found out that about 200,000 are sent per day of “private” photos. And also, that often these photos are still viewed and not complain about the sender.

Added audio and video chat messages

Over the past two years, we have drawn up features to advanced messengers and gave users the opportunity to exchange multimedia messages. Implementation of this option on the server side was quite simple due to the flexible architecture of the mobile API and its own CDN.

When designing multimedia messages in the chat, we transferred the file upload from the main client-server API to the separate HTTP services located inside the CDN. We use a single repository, so from the point of view of code support, this solution did not cause any problems. But it made it possible to use tools optimized for the nature of the load:

the mobile API (binary protocol) is sharpened by the exchange of short commands and at the peak passes through itself more than 100,000 requests per second;
CDN is tuned for working with multimedia and is capable of delivering 200,000 photos per second (we talked about CDN optimizations in detail at the Uptime day conference).

From the point of view of CDN, the type of the downloaded file does not matter much, and we already had video conversion. We could only correctly process the new types (audio and video) in the protocol of the mobile API.

Learned how to build ML-models without R&D

In 2018, we created our own framework for automating the training of models on our data. Now any backend developer can independently build a model tailored for his application, without attracting colleagues from R&D.

This framework is able to collect hundreds of metrics, both universal (gender, age, activity in the application), and specific, significant for each specific model. Algorithms for constructing ML-models are flexibly configured, and graphs of the results are available for analytics “out of the box”. At the output, the framework provides a working model that responds quickly enough (within 10-100 ms) so that it can be called from PHP directly in user requests without sacrificing UX.

Last year, we integrated the framework into various parts of our application: anti-spam systems, rating the application and some others. Mail ML is one of the most successful applications: the model predicts the probability of a click on the link in the email for a specific user. The model learns from the data of other users, similar to the recipient of the email with “important” attributes, and the “importance” is calculated automatically by the framework.

After evaluating the results, we stopped sending emails with the least chance of clicking on links. Due to this we:

increased user loyalty, which resulted in improved activity metrics in the service: people do not like "hopeless" letters;
increased inbox rate in key mailers: email services like senders with high CTR.

Launched video streaming

We are always looking for new opportunities for development. When video streaming began to gain momentum, we decided to integrate it into our application. Predicting the success of the project was not easy, so in order to reduce costs, we did not plunge headlong into the development of our solution, but found a company that provided the platform and SDK with the functionality we needed.

We were required to build feature logic, client-server interaction, implement options for sending and paying gifts, provide fast moderation of content (both the video stream and comments) and cover everything with metrics. We have assembled a cross-functional team from different departments and divisions: client development, server development, billing, platform, back office, BI analytics, design and product management. All settled down in one office space - and work began to boil.

We determined the functionality of the first version (MVP) - and a month later we launched a feature in one country after another. For several weeks we rolled it out in different countries, and today video streaming is available everywhere.

Broadcast user messages on the Arbat

It was a very unusual promotion: all October, Badoo users sent messages to a huge video billboard located on Novy Arbat in Moscow. In a month, 23,000 messages were sent to us, of which 12,000 were successfully moderated and hit the screen.

A billboard is such a smart TV with a browser and our JavaScript client inside. Five-minute commercials were shown on the screen, in each of which three minutes were given to us.

Of course, all users wanted to see the sent message on the screen with their own eyes. To make this possible, we had to answer the question when each of the messages appears there.

We should inform the user about the time of showing after moderation. Technically, at this point, the message fell into our internal schedule with a clear start time and duration, which was calculated dynamically: from 30 seconds to five minutes (depending on the number of senders).

The difficulty was that the provider, for its part, did not provide the ability to control and track the time of the commercial, so it was necessary to cunningly synchronize our schedule with the schedule of the provider, constantly monitor and adjust if necessary. It helped us that we noticed (logging is our everything!) That a few tens of seconds before the next video was shown, the smart TV reloads the page. It was impossible to focus on this time, but on it one could notice shifts in the schedule, which sometimes reached 15-30 seconds. We set up the monitoring of shifts on our side and at the moments of operation we started to request the time of the nearest show from the provider.

In the process of setting up this system, our engineer, who worked from a cafe opposite the billboard, ate an impressive amount of croissants and drank many cups of coffee, but thanks to him, the broadcast of user messages went smoothly and there were no serious incidents.

Technical projects

Despite the large volume of grocery work, we devote a lot of time to technical projects. Most of them relate to the performance of our applications, as well as the development and improvement of internal tools.

Liveprof Open Source Project

We have long had the opportunity to enable XHProf on production for a specific request, but this is not always enough. Requests are different, but for systematic work on performance, I want to see the big picture for all requests. Therefore, such an idea was born: launch XHProf automatically on a small part of the queries, and aggregate the results. This is how the Liveprof project appeared , which we put in open access.

In the process of processing the request, there is logic that, based on the probabilities and the API method, decides whether to start XHProf. Profiling results are written to disk, and then delivered to a separate server, where they are placed in raw form in MySQL.

Once a day, a script is run that aggregates the results for each brand, platform, and API method. Thus, we can see the results of profiling for specific requests from Badoo iOS (both in the form of a tree and in the form of a flame graph), we can see what percentage of the cluster it takes to call an individual function (for example, how much it costs to collect a URL for a photo), and recently added the output of this information directly to PhpStorm.

Migrating to PHP 7.4

New versions of PHP are pleased with the performance improvement. Over the last two transitions (from 7.0 to 7.2 and from 7.2 to 7.4), our department was responsible.

The transition begins long before the official release. First, we manually run the tests with the new version and ... we face a huge number of problems. Some of them are easily solved, while others require some time.

When most of the problems have been resolved, we switch to the new version in the developer environment, for some time we monitor error logs and collect feedback from developers and QA engineers. By the time of the official release of the stable version, we are ready to test on production: first we run it on one machine and gradually lay it out on the others.

But after the new version is distributed on all servers, the transition does not end: we continue to check the syntax with both versions of PHP (new and previous). Thus, we are convinced that the developers did not start using the new syntax and that we can switch to the old version at any time if unexpected problems arise in the new one. For example, before the New Year holidays, we discovered a small problem with a memory leak in PHP 7.4, decided not to risk it and returned to the previous version before the holidays. After the holidays, having dealt with the problem, we again rolled out version 7.4 and still use it.

Go aggregation of requests

With the PHP update, our struggle for performance does not end there. We have a powerful framework that does a lot of work at the start: from initialization to various checks required for each request.

We decided to conduct a simple experiment. We selected a part of the commands for which the server’s response is not critical (for example, sending statistics from the client or choosing the “No” option in the swipe game), and wrote a service on Go that accepts such requests, saves them, and then sends them to the server in a packet ( for each user). Thus, we can very easily save on initializing the application without rewriting the main code.

The experiment turned out to be successful: according to our estimates, we can save a few percent of the CPU (which in our case is more than ten servers) without rewriting the main code, but at the cost of complicating the operation and the logic of work. To complete the picture, we decided to wait for the results of experiments with PHP 7.4 preload and RoadRunner in order to choose the optimal solution according to the ratio of winnings and complexity of implementation.

Photo optimization

User activity directly depends on some technical projects. For example, the faster we show photos, the more actively people swipe, the more matches happen, the more new pairs form and so on.

In October 2018, one of our brands, Bumble, decided to enter the Indian market. Since we did not know anything about the speed of the local Internet, we decided to conduct a series of experiments with the quality and size of photographs.

We launch all technical experiments under A / B tests. This allows you to see not only the difference in the behavior and activity of different groups, but also statistics on the operation of the application on the client, broken down by options. We were interested in time to download pictures and time for decoding and display.

Infrastructure, in addition to storage servers, we allocate a cluster of photo caches - these are the servers that store the most commonly used pictures. Since there are a huge number of smartphones with different screens in the world, we store photos in several basic sizes and convert them on the fly to the size and format requested by the client.

The task we wanted to solve was to choose the optimal format (JPEG, WebP, PJPEG), image size and quality. In theory, the WebP format should be the best, but it is the most expensive in terms of conversion on the fly: instead of saving traffic, you need to buy more photo cache servers. Along with this, we decided to test how CDNs in the region will help improve UX.

Several of our engineers were sent there to launch a project in India. So we had not only dry numbers, but also people who could test the application on different operators on the spot, compare its work with the work of other applications, and so on.

The results of the experiments allowed us to choose the best options for the quality and size of photographs. For example, we saw that the logic of preloading photos on the client plays a large role in user activity. And despite the fact that photos in WebP download faster, the positive effect was noticeable only in mobile browsers. The same story with CDN: we tested three external CDN services and, despite the acceleration of content delivery, did not see any positive changes in activity. As a result, we turned on WebP for mobile browsers and saved on power, leaving all other clients in JPEG format.

Own video streaming

As I mentioned above, the first version of video streaming was launched using an external service. The experiment was considered successful - and we began to prepare software and infrastructure for launch in our data centers.

With the help of ready-made transcoder (for squeezing the video stream from the user) and Edge-servers (for distributing the stream to the user) we had to build a system similar to an external boxed solution. And, of course, a lot had to be finalized with a “file” on the go.

For example, initially each Edge server connected to each transcoder server, which complicated scaling and increased internal traffic. We wrote client balancing in such a way that clients of one stream go to the least number of Edge servers, taking into account the popularity of each particular stream.

Another example, when we had to be quick-witted, is the logic of changing the quality of a stream in WebRTC: due to the fact that we use two quality options, the standard algorithm was demolished and it worked only towards decreasing the bitrate. I had to go deeper and edit the algorithm used inside the WebRTC library.

In addition, WebRTC signaling, by default, passed too much information about hosts on the network to the client. The solution was a proxy server on Go, which gives the client only what he needs to know, and in addition collects a lot of useful information about connected clients.

From the start of research to full implementation, the project took more than six months, but as a result, we were able to significantly reduce costs by abandoning external services.

Performance optimization

This is not a one-time project for our team - this is an important area to which we also pay a lot of attention. Our main cluster serving requests from applications consists of hundreds of servers, and therefore it is profitable for us to invest time in work on optimizing the application: each percentage won costs us several physical servers.

We constantly monitor the total load on our cluster and the load for each specific API call. If a certain threshold is exceeded, we begin to search and optimize bottlenecks. The larger the excess, the more people get involved. This approach is fully justified: in the past two years, the cluster growth rate has been half the rate of growth of the audience of our services.

We actively share knowledge gained during optimization work at conferences and meetings. More details about our projects and studies can be found in articles and reports by Pasha Murzakov:

Control over old and unused code

We have quite intensive development, and the pace of launching product tasks and experiments is also growing as the company grows. Under such conditions, the code base becomes more complex and less controlled. The situation is exacerbated by several platforms and users with older versions of applications and operating systems.

We try to keep our code base “clean” in order to maintain a high pace of work:

automatically create tasks to remove the code after the completion of the product experiment;
automated tracking of old versions of applications and operating systems and the code branches used in them;
Introduced tools to automatically search for unused code.

You can learn more about our results at the Badoo PHP Meetup , which will be held this Saturday, February 15th.

Of course, these are not all the projects that I want to talk about. Over the past two years, we have struggled with spammers and snappers, sawed our plugin for PhpStorm, tested various KV repositories (and chose Aerospike), experimented a lot, shared knowledge in articles, at conferences and not only.

But if you decide that we are only working, then this is not so. Each team has its own team building budget. Our colleagues visited Amsterdam, Edinburgh, Prague and other cities. And last year we organized a general meeting of the department in Croatia:

And we also know how to photoshop.

On moving to London and the office in Soho we already told, therefore we will not dwell on the details in this article - just show a couple of photos.

Big photo

Hiring event is the easiest and fastest way to join our team. We accept answers to the test until March 1 inclusive. Then we will take a week to analyze the results and call those who coped with the tasks.

If you want to get to work for us in London, but don’t want to take the test, there is another option: just respond to vacancies on our website , this opportunity is always available.

If you have any questions, feel free to ask them in the comments or send me private messages.

Good luck

Offer in London in one day: how to get it and what to do after moving