🔞 👨🏼‍🌾 🧑🏽‍🤝‍🧑🏽 Why Masterhost was doomed: an inside look 🐰 🎗️ 📊

This company has long been in a state of slow fading. The catastrophe that occurred in the past few days is simply a truck that shot down a person with a cancer at the last stage.

I worked there for 5 years, and I want to tell you what exactly was wrong with this company.

Catastrophe

Accompanied by armed guards, Yevgeny Borisovich Benevolensky entered his data center in the style of a medieval castle on Altufevskoye Shosse, barricaded himself from the police, and, flandering between the racks, began to pull out wires.
Masterhost monitoring operators were located in another part of Moscow. They watched powerlessly as section by section shut down critical services. And soon the monitoring itself went out.

The entire IT infrastructure of one of the largest hosting providers in Russia has gone into oblivion. The attendants went home: they simply had nothing to watch and nothing to fix. In just an hour, Evgeny Borisovich managed to arrange one of the largest blackouts of the Runet of recent years.
From the side, the disaster that happened with Masterhost looks like an accident. It seemed that on March 2, 2020, the former owner did not enter this clumsy-designed building, had he changed his mind to take revenge on his former partners - and Masterhost would have happily served the website of Roscosmos and the page of the St. Petersburg Choral Synagogue .

However, anyone who saw this company from the inside knew that it was doomed.

Time of hope

In 2015, when I first arrived, my colleagues seemed to me demigods. I could struggle for hours over the problem - and then Unix specialist Petya Popov sat at my computer, leaned back in his chair and, looking at the ceiling with Buddhist serenity, typed in a ten-finger method a long command that did not fit on one line.

And after pressing Enter, the problem has already been solved.

They knew and knew everything. How good they were was especially noticeable against the background of my modest knowledge of Linux systems. But experienced system administrators taught me all the tricks: working with the console, creating scripts on bash, working with nginx, apache2 and mysql.

While I was getting acquainted with hosting monitoring systems and c software that allows you to centrally manage the configuration of operating systems, it seemed to me that I was learning to pilot a flying saucer. Puppet and ansible, virtualization and backup systems, the art of diagnosing and solving problems on highly loaded servers - I felt like just a Star Wars hero.

And when I delved into the business processes of the company, it just seemed to me, who had just recently left the Barnaul civil service, that Masterhost was a Russian analogue of Google: everything was so wise, thought out and futuristic there. Judge for yourself.

The core of the company is the technical department. It consists of five departments: Unix, windows, programmers (mainly Perl), networkers and maintenance. I, as you already understood, worked in operation. Unix is mainly engaged in hosting and mail services, maintains infrastructure, without this department Masterhost does not exist. The Win department is a separate domain - hyper-v virtual machines and mail on Exchange, which Masterhost successfully sells. Own corporate mail also depends on them (if they fall, it will be unpleasant, but not fatal).

Perl deals with billing and financial logic. This is a personal account, cash withdrawal from customers and the use of services requested by customers. Networkers, on the other hand, optimize the network and control both internal and external traffic.

The operations department is the third support line (1st is the call center, 2nd is support). Through us, all departments of the technical department interact. We solve both client requests that support throws us, and the technical tasks of the remaining departments. In short, we do everything to which we have access and knowledge. If this is not the case, we formulate specific questions and write to the senior department what we did to solve the problems (diagnostics, attempts to solve the problem) and ask for help.

In general, everything is super. And given that the company had clear career prospects, it was a dream job.

But there was one small detail. Despite the fact that most of the hosting worked on Debian 7, in 2015 for some reason there were also services on the long-not updated FreeBSD 7 and 8. And the people in the UNIX department who would understand FreeBSD 7 and 8, for some reason did not have.

That was weird. And when I began to find out the reasons, I felt like a hero in a horror movie, which, having entered a new house, found a spot of blood on the wall that was not drying. Or rather, the hero of "Moon 2112."

They just disappeared

Once upon a time there were FreeBSD experts at Masterhost: they deployed the infrastructure from scratch and supported it. But once, when the system was already debugged, they simply disappeared - immediately by the entire department. And precisely because they disappeared, and vacant places appeared, which in 2015 I and other newcomers took.

At the interview, of course, no one mentioned this nuance. It became clear later.
From the operation department, we quickly recruited the most sensible, upgraded to the admins of the Unix department and set the task: by all means prevent blackout. Since there were no FreeBSD specialists left in the company, problems could start at any moment. After some debate, the hosting decided to re-host on Debian 7. All the old achievements were lost, time and money were lost - after all, the work had to be done on thousands of servers.

If before there was some kind of strategic plan for hosting development, then from that moment up to the siege of Altufevsky castle exclusively tactical tasks were solved.

So, in February 2016, support for Debian 7 ended, which was replaced by FreeBSD, and it was urgent to move to the eighth version. But this problem is still being solved: now, in 2020, 90% of Masterhost's servers are still working on Debian 7.

Our hosting, as they say, is already dead by 2016. For a long time there were no security updates for packages: only packages that were manually rebuilt were updated. There was no need to talk about any security here: what is security without security update?

In general, the infrastructure was left to chance: the shadow of the UNIX department that disappeared in 2015 hung over us after 5 years. But where did they go?

60 months before the end of the world

Colleagues from the Unix department just got up and left. This happened cyclically: every two years, from half to two-thirds of UNIX system administrators simultaneously disappeared, after which a scatter of free vacancies suddenly appeared on the headhunter.

During my five years with me there were three such epochal departures of key employees. Most admins were not really IT demigods. These were hastily recruited beginners. Why? Did the company have no money? Or did the management simply know nothing about some malicious HR manager who did not understand the specifics of the company, or an eccentric executive director? .. No. It knew.

Evgeny Borisovich Benevolensky himself - one of the key figures of the company - explained it this way. Masterhost is not a company, but a university. And we are not workers, but students. From which it naturally follows that we are not entitled to a salary, but a scholarship. And when we are ready to graduate, we simply quit. Hence the almost zero requirements for the device in the operation department, in which future senior administrators are forged.

At one time, it was even practiced to recruit people to the department of exploitation under the so-called student agreement. This is when you are taken with zero experience with a salary of 20k, and you do not have the right to quit for 2 years, otherwise you will pay an indecently large amount for "training in craft". A sort of modern voluntary slavery. In addition, there was not a single case when I tried to persuade a UNIX colleague to persuade me to stay, to offer to raise my salary. I have repeatedly watched my colleagues and friends quit, who were tired of working for a small price list, with great responsibility.

For this reason, in Masterhost, in fact, there was no team lead. As soon as a person felt that he had become a professional, he immediately went to another company.

In general, the management understood everything perfectly: it was a deliberate strategy. High salaries are not needed, team leaders are not needed, it works - okay. In this company, in principle, there was no option to increase salaries: you either work or quit.

But maybe this is not such a bad business strategy? .. The hosting did work. Customers came and brought money. Or not?

Customer out of habit

In the early 2000s, people came to Masterhost with their sites - and their sites just worked. Clients were provided with their own e-mail, right there in their personal account it was possible to register their own domain. Then it was all new and very convenient.

There were almost no competitors, Masterhost was really cool, he gave people exactly what they wanted to get.

However, over time, there were a lot of domestic hosting companies, and everyone got about the same services, only at significantly lower prices. The services of Masterhost rose in price, but the quality remained the same. Clients began to come less and less, and old customers kept on simply because moving to a new site was technically difficult, the client did not have a specialist to transfer the site, it was necessary to look for a new site, to agree ... When everything works, why touch it?

As a result, the customer base consisted of people who stayed with the company just out of habit. Large customers, including state ones, who are too lazy to change something. Masterhost year after year exchanged its primary reputation capital, earned back in those days when monitors were convex, and Aport was the most popular search engine. Since that time, both Roscosmos and the St. Petersburg Choral Synagogue were hosted there.

But even if Masterhost wanted to change and tried to become a progressive IT company again, it would be very difficult for him.
Because he, in general ... was not.

Do you see Masterhost? But he is not!

In fact, no Masterhost as an integrated company existed - there was a group of companies and many co-owners. Masterhost is 5 different LLCs. So, a third of the UNIX department worked at Telematika LLC, another third at Masterhost LLC, and the rest at Informcentr LLC. At the same time, everyone was sitting in the same room.

The company's architecture did not imply the ability to scale and change. It was tailored solely for tax optimization, and not for the optimization of business processes. It was not a flying saucer from Star Wars, but something like a sprawling, loosely connected raft.

In theory, the technical director should bring ideas to modernize the technical part of the company, offer new services to the court of shareholders and owners of the company, but alas - in five years they have already changed 4 pieces (the one that lasted the longest, leaving, said that the company would last no more than six months). And not all of them wanted to hang out with the leadership, proving that without changes and without new services the company is increasingly lagging behind its competitors.
People came with burning eyes, but soon, when the first rainbow impression came down and they realized where they were, the enthusiasm faded away.

Summary

In general, Evgeny Borisovich began to pull out wires from this business long before March 2, 2020. The masterhost was doomed, and blackout only accelerated the process.

The conflict of ownership is just one of the consequences of legal chaos when a company consists of a dozen different LLCs. Moreover, the consequence is not the most dangerous: the intricate structure did not allow this business to scale freely.

Poor structure, lack of team lead, salary policy in the spirit of an Owl-effective-manager, inability to change, dependence on state customers - all this was an inevitable consequence of the fact that the company was not focused on development, but on optimization of profit.

It may be good that a blackout occurred. Perhaps this shake-up and loss of the old customers that made up the backbone of the clientele will force owners to rethink their attitude to business - and perhaps this dying company, with which I have so many nostalgic memories, will be transformed and become modern and successful again. After all, now she has no choice: it remains either to change or to leave.

But if Masterhost chooses the first option, he will surely succeed. After all, there always came people whose eyes burned with enthusiasm. And I know this for sure - after all, I myself was once among them.

PS Many thanks to Misha Zobern, who helped bring my scattered memories into a single whole.

Why Masterhost was doomed: an inside look