🕴🏽 ↩️ 🧕🏽 When code becomes legacy and how to live with it 🧑🏽‍🤝‍🧑🏽 🌅 🔥

Many years ago, I came to one legacy project, which was developed by Vladimir Filonov (pyhoster) So I met one of the organizers of MoscowPython, a lover of delving into the insides of libraries, and then talking about it. It is ironic that now he is going to tell how to survive if you got legacy. This once again proves that even those who later teach how to live with it generate legacy. I really want to ask Vladimir about what legacy is, how to grow less for them, how to fight when they are already legacy to the ears, and when to drop everything and write again (spoiler: never).

But first, watch the video to feel all the pain of immersion in legacy ...

- Let's first determine: when does the code become legacy?

In an ideal world, the code should gradually evolve - the company grows, the requirements grow, and the code must change, refactor to the new requirements. But in practice, most often to the once written code, new features are simply gradually being rolled up. Typically, a business is reluctant to set aside time for refactoring, if at all.

In fact, legacy begins to appear almost from the first months of the life of the project, if it has a fairly active code base. It's not that no one thinks ahead of the full architecture of the code and the system in general. It is simply impossible to do if you are not developing something as typical as an online store. But they no longer write code on them, they are made of constructors.

For everything that is more complicated than an online store out of the box, they form some kind of primary idea, somehow implement it, and then it turns out that the world needed something a little different. The code begins to be modified, but not all, but only the part that directly touches the world. The winding of the adapters begins, something else. Already at this moment appears legacy.

At first it seems that this legacy is quite painless, it is easy to refactor later, but for now you can arrange TODO, perhaps even make tickets for future refactoring. Such tasks gradually accumulate, and the concept of technical debt appears.

If you try to mark the point of origin of legacy on the project’s life schedule, then I would say this: as soon as you look at the system, you get the feeling that there is a technical duty, then legacy is already here.

- You raised an interesting problem that an incompetent manager either allocates very little time for refactoring or does not allocate at all. Why do you think this happens?

From a business point of view, product development is an increase in functionality or a change in requirements, the generation of artifacts for the business itself. And refactoring in the first approximation does not generate anything . Even if the existing system is written haphazardly, it somehow works, sometimes it crashes, but it brings a fair amount of problems to the business. It is not clear to business why to refactor this code and what it will give it. The answer is that in the future with development this will help to avoid problems, the business does not satisfy. It is difficult to see the problem where they are not yet.

Few people are interested in future problems: if it falls, we will sort it out, it will start to slow down, and we will redo it.

The business does not understand the argument to reduce risks. It will start to slow down, which means that there will be more customers and money, which means that it will be possible to buy more servers. It is difficult to convey technical aspects, for example, that far from always horizontal or vertical scaling will give an increase in productivity. In addition, most developers are not able to speak the language of business, then they generally can hardly convey something. As a result, the two departments speak completely different languages about different things. There is no way for them to agree, and therefore justifying that refactoring will give something is often very difficult.

- In contrast: what do you think, how does a competent manager relate to this issue? What is the difference between the two extremes?

The competent manager, most likely, already had experience working with legacy systems and the consequences of the code base becoming obsolete, raking in problems, including a drop in sales, because the site suddenly started giving in five hundred times more often, and customers couldn’t reach to the final stage of payment, for example. Or simply did not wait for a response from the server and went to another online store.

If the manager once felt these risks (or repeatedly, if he does not learn well from mistakes), he no longer needs to explain anything. When they again say magic words that he didn’t listen to last time, for the first time out of inexperience, he will most likely begin to go too far in the opposite direction. And the development process will turn into constant refactoring, races will begin for obscure indicators of code cleanliness, test coverage, etc. ... In general, the balance will be shifted to the other side.

And when the business, on the other hand, comes up against the fact that development has stopped, it will come to an understanding that refactoring is just a routine process, it needs to be incorporated into the development process. If the company is in one form or another agile, then some amount of work with technical debt should be included in each sprint.

Refactoring is an essential routine. When it is included in the development process, there are no problems or cost overruns for emergency treatment of accumulated technical debt.

- Indeed, this is probably the most working option. And what are the practices, tools and approaches in order to overgrow legacy as little as possible?

From a management point of view, this is an iterative development . In planned development, for example, when Waterfall is strongly planned in advance, it is difficult to identify the points when and how much to refactor, because it is not clear how many compromises were made during the development process. In the iterative case, it’s simpler: every time we see how the sprint went, there are retrospectives, and you can evaluate the very technical debt generated by previous sprints.

- Since you said “evaluate,” let's discuss, and how to evaluate technical debt?

There is a test coverage, which, of course, does not directly relate to code refactoring and obsolescence, but refers to the quality of the code. It is clear that in an ideal world code coverage with tests should be 100%: the developer writes something and immediately completely covers it with tests. But in practice this does not happen, far from everything is covered by tests, moreover, there are different tests for tests.

We all know that 100% unit testing doesn't prove anything yet. Especially in the modern world where microservices walk, integration tests, end-to-end testing are needed. This is precisely done separately from the development process. Therefore, it is possible, for example, to evaluate the increase in functionality not covered by end-to-end testing, in which parts there are manual tests and no integration tests. This is the first thing that can be evaluated and what to start tickets immediately as a technical debt.

Secondly, you still need to invest more time in design . When making new functionality, a new microservice, you should try to use the API first approach - first, prepare an interaction scheme (as an option, documentation in Swagger), and do not start writing code. This will reduce the amount of legacy, because it will first understand what should be in the output, and then to this understanding begin to implement the code. Most of the compromises that you need to make will be resolved at the level of discussion of this contract. Future dependencies on other parts of the services will be foreseen immediately, and entrances and exits will be laid for them.

When everything is done directly in the implementation of the code, most often it happens like this: now we’ll stick a crutch, because the deadlines are on, you need to finish the feature in this sprint. If design is first done and problems are discovered at the design level, they are resolved before they start writing code and before they give a final assessment of the labor and time costs. Many problems can be resolved at the design stage, and less workarounds will be required afterwards.

If we return to the tests, there is a well-known approach Test Driven Development . I love him very much, but like any tool, it has its own scope of application. I would say that it is really useful to use Acceptance test – driven development: when we describe the functionality, we deliver it with top-level tests that check how it should work from the point of view of the business at the top level, without touching the implementation. The process of formulating these tests will help to better understand the problem itself. The final developer, including, will have a reference point, whether he understood the task correctly.

This is one of the moments when legacy appears: when the task is formulated inaccurately or in the wrong language that the developer understands.

The developer didn’t understand something a little, did something a little wrong, the release time approached, it turns out that the feature does not work exactly as it should, it is quickly filed with a file to what the business expected. With Acceptance test-driven development, the risk is reduced that we will not cut what the business wanted, because there are tests that show what was expected.

- We have already touched on this topic in several issues, but they talk about it much less often than about problems with the code. How to transfer the knowledge that once was somehow reflected in the code, but naturally, not all with distortions? What if there is not a single person left in the team who knew what, where and why was done?

This is really a very big problem. In management, there is even such an antipattern SHOK - Single head of knowledge. This is one of the main problems with legacy, which I want to talk about in a report on Moscow Python Conf ++. Many troubles are caused by this very mistake, when one person who developed everything himself and understood everything disappeared somewhere. In more or less mild cases, it partially disappeared, for example, received an offer and moved to another country, but sometimes it responds to e-mail. But there are cases that are really painful for business, when such a person did not say anything to anyone, does not answer questions, and generally does not care.

There are two sides: the first one that a business should think about, the second one that a developer should think about if he plans to be a good developer. Business should formulate approaches to preparing and accepting work. Two simple mechanisms will work: Definition of Ready and Definition of Done .

First, we agree on how the task should be set in order to consider it suitable for development. This helps to figure out what the business really needs, because it happens, the customer asks for one thing, but in reality his pain can solve something completely different. And, of course, competently composed tasks are then more convenient to read. If a new manager arrives, he can read the tickets of the last few sprints and understand what and how has developed lately from a business point of view.

Definition of Done, in turn, on how the artist then accepts the work. Unfortunately, often management does not understand how to do it right. Suppose a developer comes, says that he has finished work, shows something, it more or less looks like what he was asked for. But in practice, everything inside can be done poorly or even well, but not documented, no tests, etc. Definition of Done helps to accept work simply by using a list: for example, if there is a ticket like bug, then you need to look at the playback steps, check if there is a regression test in the CI system, etc. This, of course, is already a pretty good case when there is a CI system and there is such a manager.

But even when everything is much worse, and there is only a startup manager from management, there is no CI, there is nothing, and there is only one programmer from the executives, Definition of Ready and Definition of Done help in the same way, they just become simpler.

Definition of Ready and Definition of Done in terms of knowledge transfer is a very convenient mechanism. It allows you to properly share information.

A programmer takes a task only when it meets the Definition of Ready formally and according to his subjective impression, and the manager accepts the finished work when it formally and subjectively corresponds to the Definition of Done. If one or the other side at the time of acceptance of a task or result is not objectively or subjectively satisfied, a dialogue begins, during which all that is needed is specified.

It is very important to conduct all these dialogs in the same tickets. That is, if tickets are placed through a Google document, use comments; if you correspond by e-mail, write everything via e-mail. Never change the mechanics. If you discussed something over the phone, be sure to write down follow up, otherwise this conversation will be lost. You can even record conversations and upload them to the cloud. Modern communication technology allows you to capture anything. The only thing that prevents is the lack of organization.

It is necessary to add a little organization to the process and the transfer of knowledge will already take place. She will not be perfect, but she will be. Because the biggest problem, when the developer has disappeared, a new one comes, asks: “What do you have here?”, And no one knows what they have. But if there is an entire story, even if extensive, not very ordered, it may not be quick to figure it out, but you can still understand something.

If resources allow, there is time, the ideal option would be to introduce a wiki and accept it as an internal regulation. This, by the way, also refers to the description of Definition of Ready and Definition of Done. If we are talking about microservices or even any systems with an API, then the API must go through Swagger, that is, with documentation, description, models. Then the API itself becomes the object of documentation.

In general, the load on the developer does not increase significantly, because in all modern frameworks there are libraries that allow using the same decorator to add documentation for Swagger auto-generation.

Plus, starting with the latest versions of Python, type annotations have appeared that also allow a lot. I have not yet seen tools for generating documentation on type annotations, but this allows you to compare code and annotations, and it also does not take so much time if laid from the very beginning. If you introduce it later, then you need to start tickets for technical debt in this area, and gradually, and not with a snap, try to cover everything with type annotations.

- For many years, the question does not leave me: why all the instruments that enter the market do not adhere to legacy-first? MyPy assumes that everything is good in the code at once, and if everything is bad for you, then everything is bad for you, do not try to use MyPy. Why do you think people are not legacy-tolerant?

This question also interests me, since in legacy it is always painful to introduce the mechanisms of modern tools to increase the readability and documentation of the code. It seems to me that this largely comes from the fact that most developers have a subconscious and sometimes conscious opinion that any legacy is easier to rewrite from scratch. This applies to any project with quite extensive legacy, where these pains are really painful. Of course, if we are not talking about a very tiny legacy, then MyPy is not difficult to implement there either.

Due to the fact that the majority believes that it is necessary to rewrite everything, since everything is already bad and there is no need to try to fix it, this attitude appears that if you have extensive legacy, then large convenient tools will no longer help you. I do not see other reasonable explanations.

- Hence the next question, but how to really understand whether it’s better to rewrite or can be saved?

One of my colleagues once dreamed of a certain "point of Kirch." He didn’t remember at all what this meant, but we liked the expression so much that we began to wonder what to attach it to. And in the end, they came up with the point that Kirch is the point in the project when an attempt to revive a dead horse becomes more expensive than an attempt to rewrite everything from scratch.

In practice, its definition is an unsolvable task for one simple reason. If you have a fairly extensive system with a lot of old code and old solutions, then on the one hand it is very difficult to modify, update, etc. On the other hand, rewriting from scratch will lead to the fact that it will be necessary to create a system of the appropriate size, which means that a new legacy will appear on the road one way or another.

Moreover, if you have in your hands a system that has been developed and has already worked for a considerable time, then the developers went through a variety of rakes related to both business and technology. Since this part of legacy is most often not documented, when rewriting, you will have to go through this rake again. Most likely, the solution that will be at the end will be comparable: either it will be the plus or minus is the same, or it will take much more time.

You can prove to the business that you need to rewrite something, if you convince yourself that it will be easier than supporting it, which means that you will be severely limited in terms of time, and in the end you will bring about the same thing. Therefore, with rewriting you need to be very careful.

I think completely rewriting never needs anything. The only reasonable option that I see is gradual substitution.

If there is a large monolithic system, and which can no longer be maintained, it is worth looking in the direction of switching to multi-services. I do not say micro, because they can be very rather big and there can be few of them.

Instead of upgrading part of a large old system, you can make a small subsystem on the side that implements the same functionality. Locking on the old dependency in terms of the web server, application server, database, we first implement in the middle, at the application server level, another application, using the web server we send part of the requests to the new service. But it can continue to work with the old database if it stores inalienable data. If alienated: they do not have hard dependencies and have weak connectivity, then you can transfer the data to a separate database.

At the next stage, when it will be necessary to upgrade something else in the old system, another service is possibly generated, otherwise we simply expand the functionality of a new small service. But now we do it by all the rules, carefully work with the new code, and then we will not encounter the problem of the lack of legacy first in the tools. So we can work with the new code, as, for example, MyPy requires.

It seems to me that this is the only option for business and for developers that can be used to remake a large old system - not to rewrite, but to remake, gradually replacing its individual parts and not falling into the trap.

Because if you try to rewrite everything directly in the same code, you will inevitably run into the problem that if you get into a certain code, you get stuck there anyway, because you are pulled not only by purely external dependencies such as databases, but also dependencies on classes, from local models and technologies. If the code is written in Python 2.7, you can’t take it and partially translate it into Python 3.

- So we come to the next important topic: what if not just the code became legacy, but the platform itself became legacy? How does this change techniques for working with legacy code or with a project?

If the platform has become legacy, then this is a big signal that we need to stop developing the old code. Support is sure to remain, but to develop in terms of growth of new features you need to finish. And so, use an iterative approach with a partial replacement with new services or a new service, which then grows, but already on the basis of new modern technologies. We can take the old system in Python 2, and when you need to redo the section, do it already in Python 3. Naturally, the whole binding around will remain. Or the part that can also be transferred will not remain. For example, if it's not just Python 2, but Python 2 on PostgeSQL 7, and everywhere PostgreSQL 12 is already needed.

At Moscow Python Conf ++, stories about how someone switched from the second version to the third, a decent amount. But I'm not sure that all these stories are not the survivor's mistake. It is not known how many projects they tried, but did not succeed and rolled back or didn’t roll anywhere. I'm afraid it’s like with startups - we see thousands of successful ones, but we don’t see hundreds of thousands of dead. In addition to some, whose founders find the strength to speak at Startup Fail Night and talk about how epic they failed. This also happens with projects, but almost no one talks about it.

- Do you think the programming language itself is an important factor on which the legacy project will grow?

I think an important factor is not so much the language itself as the ecosystem. The Python developers really did a lot to help the developers switch from Python 2 to Python 3. For example, when the third version just appeared, a special tool 2to3 was released, which, of course, does not solve everything, but helps.

The transition from second to third Python was very lengthy. For how many years there was version 2 and version 3, and the second was pretty well supported, and some features were ported there. The developers were given a lot of time to solve this problem. From a community perspective, I think it was a very good process. We were warned in advance that it was not necessary to start a new one on the old version, that Python 2 would cease to be a supported language, it was time to switch to Python 3.

For many years I have not seen new projects that would choose a second Python at the start. So this mechanics worked.

But, on the other hand, Python is a language with quite a lot of batch support. There are many ready-made libraries that developers actively include in their projects. And it often happens that some of them subsequently do not migrate to the new version, that is, they are not adequately supported. And you, for example, have a half-system on them, and you are nailed to this framework. And all - you are doomed.

This is exactly the situation when you need to gradually switch to new frameworks, and there is nothing to be done, because you can’t expect that everything will migrate.

That is, of course, the ecosystem affects one way or another. I'm not sure if there are ecosystems that solve such things more transparently. But the ecosystem with tests is still better built in Ruby and, they say, in Go. To be honest, I myself worked with Go very little and can’t say that I don’t know much in practice.

And due to working circumstances, I also had an incredibly unexpected experience with Clojure. This is a very academic language, compared to Python it has a very high entry threshold. In Clojure, the quality of engineering approaches is initially higher, simply because developers are higher level. I mean, the engineers who work with him, most often already with some experience. The June will not go to Clojure, and if they do, then I am very sorry for them.

In general, Python is not so bad.

- And what was the most wild thing you saw in the code?

One of the wildest things that Python has, it’s like a ghost, wild, but cute - it’s metaprogramming. It sometimes gives rise to monstrous homunculi, those same chthonic monsters. And when they are not documented, it becomes a hell of a hell: you see the code, you read it, but it is not in runtime. That is, something quite wrong happens in runtime, and you won’t find the end, because the meta-class, meta-class, meta-class on the meta-class, they all give rise to each other, and it’s not clear what is happening at all. Your only friend is runtime debugging.

There is still a lot of fun code going on when, for example, a former Java developer writes in Python. You come to such a project and you see essentially Java, everything is completely not the Python way. Now, many more JavaScript developers are starting to write Python code for some reason and also commit outrages.

- I collect similar examples in the python-code-disasters collection . By the way, you can add your own code to it to warn others against errors.

At the conference, at the stern of the report, I will conduct a workshop for which I will definitely find some kind of scary-scary code. I will use the power of community to search for it (if you just have such examples, write in the comments to the article or to me on facebook) and honestly I will not watch until the conference. At the workshop, I will open the legacy code for the first time, and in real time I will try to make changes so that it is guaranteed not to break anything. The task of the workshop in practice is to figure out what's what. The report will be more theoretical, and even, perhaps, captain's, just backed up by extensive experience. But besides the story about how not to get into legacy or how to survive when I’ve already gotten into it, I want to share how I parse the legacy code, because there are completely portable tools for this in my arsenal.

Of course, the sixth sense helps me in part, but in part it’s just the skills of how I look at the system that I see for the first time, in what order I disassemble it, what notes I make to myself when parsing. At the workshop, I’ll try to build a more or less sane picture or a graph of how the system works so that other developers can work with it in the future.

- Since you started talking about the conference, tell me, you were a guest and a community activist, a member of the Program Committee, and now a speaker: what changes when you attend a conference in different roles?

This is a very different experience.

When I first went to the meeting, it was the very first Moscow Django Meetup. It was 2012, by that time I had already been engaged in commercial development for 9 years and was not at any event. I didn’t go anywhere at all, talked only on the network and saw other developers live only in the office.

When I first came to the mitap, I was struck by the feeling that around me there are people who have the same pain that I have.

But only some have already passed this pain and can tell me how they coped with it. And others hurt that I hurt half a year ago, and I have already treated it and I can share the recipe myself.

That was what the English name is Wake Up Call - a big discovery. I realized that it is very cool when you can talk with people with similar problems on how to solve them, share experiences. Actually, after visiting the first meeting at the next, I already spoke because I saw that I had knowledge that was useful to others. Of course, it would be ridiculous and strange to watch this performance now, but I shared my experience.

Then I spoke several times, and gradually realized that my experience was being exhausted. In order to make a good report, you need to accumulate a decent amount of experience, but it accumulates slowly. For one performance, a couple of years of savings are realized.

But I wanted to continue to help people in the community, because when I started programming, there was neither a community nor normal tutorials - almost nothing. I remembered the pain when you do not understand anything, all that you have is a forum where some Indians in obscure English discuss something. I knew something, knew how and wanted to help others, so I began to help Valentin Dombrovsky with the MoscowPython community in building processes: holding meetings, organizing other events, including several years later brought Django Girls to Russia.

The first time as a member of the program committee I participated in the first PiterPy. There I discovered something else interesting and new for myself: speakers, most often, are people with much more extensive experience than you, and as a member of the PC you have a lot of opportunities to experience this experience, communicate, ask additional questions. Unlike a simple conference participant who listened to the report, he chatted a bit, and has 10 minutes of contact, which is not personal, because there are many other people around who also want to communicate with the speaker.

- That's how we have been interacting interestingly for an hour.

Communication with the program committee, of course, is important for a potential speaker, because it helps to formulate ideas more accurately. But for me, as a PC member, this is a cool experience, much deeper than just communicating at a conference. It moves forward, therefore it gives much more detailed information and, as in development, gives iteratively. I can ask a few questions, then think about it, comprehend what I heard, return for clarifications.

Most importantly, I still like to participate in conferences in all forms.

I still go to listen to the reports, because it is interesting, sometimes it clings and makes me think in the direction I did not think in the past. I still go to perform because it allows me to help someone. And, most importantly, it also structures my own knowledge, because before you sit down to give a report, you have a large amount of knowledge, shoved into completely different nooks and crannies of your head. And only while you are preparing for the report, it is put on the shelves, and you yourself begin to better understand the topic.

As a member of the PC, I like to communicate with other speakers, because it allows me to understand their issues much more deeply. Plus, by the way, this allows you to better understand how to perform well. Because we have guides for preparing the speaker, which you read and think: why didn’t I do that. Then you watch how others apply your recommendations in practice, and you understand that the next time you definitely need to do the same.

For example, this year, for the first time, I prepared an invoice with the help of Grisha Petrov for the first time. Previously, I tried to keep the entire preparatory part in my head, then from this I wrote a plan and made slides. Now I have a document with an invoice in which there are a lot of words. It is completely chaotic in its form, it cannot be decomposed into slides, but it contains all the information, and there is much more than I can remember in any unit of time. I just open it and start making a plan. It's great.

- Thank you for sharing this valuable experience, see you at the conference. Moscow Python Conf ++

Conference . , , . , -, legacy, -, telegram-, Call for Papers , -, (eyeofhell), .

When code becomes legacy and how to live with it

More articles: