Get best of the week

Why developers are so slow: common problems and their solutions

Hello, Habr! I present to you the translation of the article Why Development Teams are Slow: Common Software Jams and Solutions by Eric Elliot.



If you like listening more than reading, then in audio format, translation is available on Yandex.Music and Apple Podcasts.

Let's look at what causes disruptions in the software development process, and what you as a manager can do about it. There can be many reasons, so our list, of course, will be far from exhaustive. Instead, we will focus on a few of the most common issues :

  • Unrealistic expectations
  • Too many open tickets
  • Uncontrolled scope of tasks
  • Accumulation code review
  • Poor preparation
  • Burnout developers
  • Bugs
  • Staff turnover

Slow development is not the root of the problem. This is a symptom of the other problems listed. In 100% of cases, if the development team works too slowly, it is the leader’s fault. But the good news is that you can fix it. Let's look at each of the items in more detail to understand what we can do with each of them.

Unrealistic expectations


Most of the problems with the productivity of developers generally do not affect the development itself. It is rather a problem of our perception of the development process as managers and stakeholders.

The most difficult part of a manager’s job is understanding that writing code takes as much time as it takes and trying to speed up this process will only slow it down and increase the number of bugs. Patience is our everything.

Most often, the problem with the speed of work is not that the team is not productive enough, but that it faces high expectations. And this is entirely your responsibility. If the pressure comes from a higher leadership, then you have not formed their correct vision of the situation. If pressure comes from you, read on.

We often forget that the software we create is something fundamentally new. If you already have software that does the same, buy it, use it, import the module, etc. No need to re-write it from scratch. The new software is unique. He does something new or does something different. It is then that we create it. And since we have not done this yet, how do we know how long it will take?

Builders build prefabricated walls at the same pace, and therefore can give more or less accurate estimates based on observations. Software developers do not have reliable data to rely on. This problem is exacerbated by the different speeds of different developers, and it can vary by an order of magnitude.

As Steve McConnell, author of The Perfect Code, writes: “The conclusion that there is a significant difference between the productivity of different programmers has been confirmed by many studies by professional developers (Curtis 1981, Mills 1983, DeMarco and Lister 1985, Curtis et al. 1986, Card 1987 , Boehm and Papaccio 1988, Valett and McGarry 1989, Boehm et al. 2000). We do not have enough data to predict how long it will take to complete our project. We will find out what the scale and complexity is already starting work and this process often has many surprises. “Development is not only planning, but also research, no matter how carefully we tried to foresee everything.”
« 90 10 , . 10 90 »
— , Bell Labs

There are several reasons for high expectations that a manager can control. One of the fundamental reasons is measuring the wrong things.
You may be familiar with Peter Drucker's famous saying: “What is measured is controlled.”

And of course this is great advice. Of course we must measure! But we miss the essence of this quote, moreover, we turn its meaning upside down.

The whole idea is: “What is measured is controlled, even if it is measured and trying to control is completely useless, even if it harms the goals of the organization.”

Two examples of things that are not worth measuring:

  1. Predictive burndown charts showing a graph of the number of open tickets predicting the end date of a project based on recent measurements of work speed;
  2. The number of tickets closed by the developer, showing how many tasks an individual developer completed.

Measuring these two things cost countless companies huge losses due to loss of productivity, employees, and other costs.

Task progress diagrams


Many software tools try to predict the completion date of the project, based on the current scale of tasks and speed of work. The problem is that not one of them takes into account the unexplored volume of tasks. Moreover, this is impossible, because the time it takes to close one task can vary by an order of magnitude for different tasks, which can significantly distort the average values ​​that were calculated for already closed tickets.

If you set a deadline based on the date from the chart, then consider that you have not met the deadlines. The only thing that can save you is to throw as many tasks out of scope in the future as possible.

When you base your grades on incomplete information, you and your team will definitely pay for it. Unrealistic estimates create unrealistic expectations. This can be a real disaster if you also share these ratings with the marketing team, customers and the press.

Not all charts are evil. Those that do not try to predict the future may be useful. They can warn us about the spread of the project and combinatorial explosions when you see that the number of tickets increases instead of decreasing or moving up and down. Useful task diagrams demonstrate already realistically closed tasks, rather than trying to predict the future.

A good indicator is a curve that has ups and downs, but overall it moves down, which shows a decrease in the number of open tasks by the end of the project.

Project schedule where the number of tickets is reduced

A project suffering from an overgrowing scope will opposite be represented by a curve that bends upward.

Schedule of a project drowning in new tasks

Keep in mind that the point of observing this curve is not to try to change it, but to recognize and solve underlying problems. We do not want programmers to simply stop opening new tickets.

The goal is transparency of processes, not a beautiful downward curve.

Beware of the Goodhart principle : “If a dimension becomes a goal, then it ceases to be useful.”

Not any forecasting is harmful. When you have a hard deadline (for example, you are trying to release a game before Black Friday), you can systematically control the scope based on the average speed of work, so you know when to start trimming the scope. If the forecast tells you that you will finish no earlier than December, trust it. The time has come to prioritize and reduce.

The rule of thumb for predictions of this kind is:
“If the prediction says that you can do something by a given date, don’t believe it, if it says you can’t do it, believe it.

Tickets closed by one programmer


The idea of ​​recounting all the tasks performed by one programmer, and then comparing this figure with the average value is very tempting. But I urge you to resist this temptation. There are tons of better ways to collect developer productivity data.

There are two fundamental flaws in calculating closed tasks. Firstly, the tasks are not identical to each other in complexity and importance, and in fact, the value of the work depends on the power law. A small handful of tasks account for an order of magnitude more significance than the “average”. It's like the difference between the foundation of a skyscraper and the last nail clogged. Thus, simply by counting the number of closed tickets, it is impossible to know exactly the value of an employee.



Many years ago, I worked on a shopping basket for a global retail leader. Once I stopped writing code and closing tickets in Jira and added another ticket: “Usability study”.

I’ve been working on a basket redesign for over a year and the release date is fast approaching. Until that moment, no usability testing of the new ordering process was conducted by the user, so it took a week. We gave early access to the thousand most loyal supporters and interviewed them to gather feedback.

I analyzed the results and found an alarming trend in polls and logs: the frequency of users leaving the site at the basket stage was too high and was expressed in a two-digit number. A real disaster was coming! So I got to work and planned to record new usability tests with a personal presence. I sat the newbies behind our new basket, gave them some tasks and left them alone with the site. I didn’t say anything, I just watched them use the new interface.

I noticed that in the process of placing an order, people have difficulties with errors in the forms. Having this data, I slightly corrected our open source project on the github (without noting anything in the Gira). After some time, we conducted another test. Users have become much less likely to leave the page of the basket: the income difference for the company is $ 1 million per month.

In the meantime, my colleagues each closed 10-15 tickets. You could argue that I could just get more usability testing tickets to reflect the reality. But then I would have to make a thousand additional tickets, which would only create noise and require a lot of time.

Another reason why counting closed tickets is ineffective is that the most effective team members are also the people everyone is turning to for help. They know most about the code base, or they are great developers, or they have outstanding communication skills. They help you get through the backlog of pull requests, review the code of other programmers, train and mentor your teammates. They are the most productive in the team, because they help the rest of the team to double the speed of work. Perhaps the tickets that they close are the creation of frameworks or libraries that increase the productivity of the entire team. They do most of the work while the rest get recognition.

If you are not careful, then see the contribution of your most productive developers. The best way to find out what programmers are doing for a project is to ask them. Ask the team who they think is their most helpful colleague.

Usually the information that is reflected in this feedback is very different from the data that can be obtained simply by counting the number of tickets.

Collect performance data, but don’t approach each developer with one measure. Development is a team sport and each participant in the process plays a role in it. There is no single magical method that would be suitable for evaluating all, without exception.

Too many open tasks


It would seem simple - open a ticket in the tracker and move on. But each task in the tracker requires a whole processing cycle.

They need to be sorted, prioritized and assigned an artist before developers can begin to implement them. This work is repeated every time developers close one task and choose the next. If you have a project manager or a scrum master, they do this every time they re-prioritize the list of tasks (which usually happens at the beginning of the sprint or just once every couple of weeks).

Then the developer needs to delve into the context of the task, understand the essence of the problem, decompose complex tasks into subtasks, and only then you can finally begin to execute.

Creating and reading tickets is a lot of work, but it's like a fake job. This is a meta task. She is needed to begin real tasks. By themselves, they have zero value. And this takes time every time programmers choose the next task. The fewer tickets at one time are in the tracker the better. The less low-priority tasks hang in the backlog, the higher the chance that the developer will choose a high-priority task.

If there is a bug that only one user has mentioned, how important is it to us? He touched one person, but do we have bugs that more people noticed? Are there any new features that will be more useful than fixing this bug?
Probably yes.

Remove the noise from the backlog. Delete what you are not planning to do in the near future.
If this is really important, add later when there is time for this.

Uncontrolled task size


I like to ask developers in my team to break the work into tasks that they can complete in one day. This is more complicated than it seems, because it requires the ability to divide complex tasks into small ones, which can also be tested separately from the rest of the application.

Example: you are doing a new checkout process on a site. You don’t need to mix UI components, state management and communication with the server into one giant commit involving 13 files, all deeply related to the current code base, because the result will be a huge pull request that is difficult to review and merge.

Instead, start with an independently tested client-side basket state module and do a pull request for this. Then build the server API and make a separate PR for him too. Then write a UI component that would import the state from the module and communicate with the server API. Each of these tasks can be divided into separate tasks, although all this is essentially one big feature. As a bonus, tasks can be scattered among several programmers and speed up development, taking advantage of the size of the team.

The feature-switcher will make this process easier and safer, allowing you to turn off the developed functionality until it is ready for inclusion on production.

Note:Do not try to do this without good coverage of the smoke tests. You should be able to make sure that you haven’t broken anything by deploying semi-finished features. Be sure to check how it works both on and off.

Accumulation of tasks by code review


When developers bite off more than they can swallow, the result is a huge pull request waiting for review and verification.

This is the integration phase in “Continuous Integration” (CI). The problem is that the longer the PR remains open, the more time is spent on it. Developers will open it to see if they can help curb it. They will leave feedback, request changes and the request will return to the author to make changes and further approve these changes. In the meantime, all this happening pull request will move farther away from the master.
When several developers have a habit of making very large commits, the number of pull requests begins to grow like a snowball and integration becomes more and more complicated.

Example: Bob makes changes to a file that Jane also rules, but has not yet mortgaged. Bob’s pull request is held first and PR Jane becomes one commit further from the master. Now she can’t keep her branch until she fixes all conflicts with Bob’s code. Multiply this situation by the number of programmers who work with the same code in your project. Such “traffic jams” give rise to an excess of actions.

We calculate the number of such actions in the standard scenario:

  • Bob and Jane start work in the same branch (0 actions)
  • Bob makes changes and commits to his branch. Jane does the same (2 actions)
  • Bob's code goes to master. Jane downloads Bob's changes and discovers a conflict. She corrects it and commits the result to her thread. (3 actions)
  • Jane opens a pull request. Bob notes that her changes in his code will break something that she did not take into account. Jane makes changes based on Bob's comments and commits the code again (4 actions)
  • PR Jane finally merge. Only 4 actions.

Now consider a situation in which commits would be less, and pull requests would flicker faster:

  • Bob makes a small change and his code falls into the master (1 action)
  • Jane downloads the new version of the wizard and writes her code taking into account Bob's changes. (2 actions)
  • Since committing Jane is also small he is quickly held in the master. Total only two actions.

When we create small PRs, we significantly reduce the need to redo the code, which is caused by conflicts and code complexity.

Poor preparation


The IT industry is terrible in terms of training and support. Universities in-depth teach algorithms that are already built into standard libraries and only a few programmers write them from scratch.

While the basics of development such as the principles of abstraction, connectivity and engagement, modularity versus monolithic design, working with modules, composition of functions, composition of objects, design of frameworks and application architecture are overlooked. Due to the explosive growth of the industry, approximately half of the developers have less than five years of experience and 88% of specialists believe that training would not hurt them.

Teams work slowly because they have a poor understanding of what they are doing and no one wants to teach them.

Our task as managers is to hire experienced specialists who could guide our teams and allocate time for them to do this.

What can be done:

  • Code review: developers learn a lot by learning each other's code
  • Creating pairs of senior and junior engineers: it is not necessary to create permanent pairs, a one-time union to solve a specific problem works fine.
  • Special time devoted to mentoring: hire experienced professionals who love to learn and communicate well and give them time to share experience with junior developers, this will help the latter understand how they develop their skills.

Burnout


Making your team burn out is a big setback for the leader than failing to meet deadlines.

Burnout is a serious problem that can lead to loss of developers, staff turnover, huge costs, increased bass factor (Bus factor - a measure of the concentration of information among individual project members; the factor means the number of project participants, after the loss of which the project cannot be completed by the remaining participants) .

But, more importantly, burnout leads to health problems. The consequences of burnout can lead to destabilization of the body and even death from a heart attack or stroke. In Japan, this phenomenon is so widespread that they even have a special word: “karoshi”.

The leader can burn out the entire team, completely nullifying its productivity. The problem of burning out entire teams is especially common in the computer game development industry, where Black Friday is almost always a tough deadline.

Unfortunately, “die, but do it” is a common principle of organizing the work of these teams, although managers rarely realize the dangers of such an approach.

Instead of forcing developers to work more, managers should recognize that 100% of the responsibility for meeting deadlines rests with management, not developers.

Deadline is easier if you use the following techniques:

  • Better prioritize tasks and reduce scopes
  • Streamline processes
  • Recognize and refactor code confusion

Staff turnover


Data collected by Linkedin in 2018 showed that the IT staff turnover is superior to that in any other business. And this is bad, because it causes the risk of the bass factor, the risk that you will lose key specialists of your project.

Many companies do not attach importance to retaining specialists. Let's take a closer look at how much the staff turnover costs.

Placement of vacancies costs 15-30 thousand dollars. Engineer time costs an average of $ 90 per hour. Multiply this by about 50 interviews and many, many hours to answer the newbie’s questions and help him get accustomed to the team. Thus, we have already spent 50 thousand dollars, but that is not all.

It may take up to a year for a new employee to reach the level of the developer whom he replaced, with the first time he will make a lot of mistakes and spend a lot of time fixing them.

Thus, hiring and training a new developer, opportunity costs and loss of productivity of a team that has to train a beginner for some time and at the same time carry out part of his work is almost 90% of the salary of a departed developer. Finding a replacement can take several months, and then it will take some more time for the beginner to achieve its full effectiveness.

This is all very time-consuming and large teams are constantly suffering from ongoing turnover, because according to a 2019 survey by Stack Overflow, 60% of developers changed jobs over the past two years.

By the time the developer finally starts working with maximum efficiency, you will lose it.

How to avoid fluidity? Here are some tips from various sources:

  • Pay honestly
  • Raise your salary regularly
  • Let people go on long vacations
  • Offer remote work
  • Keep realistic expectations
  • Provide tasks that will be of interest to the developer
  • Don't let the tech stack get too old
  • Provide training and career opportunities
  • Provide Health Benefits
  • Do not force developers to work more than 40 hours a week
  • Provide employees with modern equipment

Bugs


If you think that you do not have time to implement a high-quality development process, then you really can’t do without it.

According to Evaluating Software Engineering Technologies (David N. Card, Frank E. Mc Garry, Gerald T. Page, 1978), well-optimized processes can reduce errors without increasing costs. The main reason is that according to another book, “Software Assessments, Benchmarks, and Best Practices” (Casper Jones 2000), defect detection and correction is one of the most time-consuming and expensive tasks in development.

Bugs are notorious for causing the need for code processing, and the later you find them, the more expensive it will be to fix them. When a developer is instructed to fix a bug already discovered in production, this often tears him from what he was doing. In the book “A Diary Study of Task Switching and Interruptions” (Mary Czerwinski, Eric J. Horvitz, Susan Wilhite) it says that a task from which we are distracted can take twice as much time and contain twice as many errors, which suggests that high-priority bugs are in some ways contagious: fixing one, we are likely to spawn new ones.

Bugs in production also require us to pay more attention to support and are very annoying and tiring users, which will ultimately cost you money. You will have to invest in fixing the old functionality instead of creating a new one.

A bug found at the development stage can be fixed in a couple of minutes, while a bug found in production will go through many additional phases: reporting a bug, checking, prioritizing, appointing an artist, and finally developing.

But that's not all. This bug will have its own commit, its pull request, review code, integration, and possibly even its own deploy. And at any stage some test may fall and the whole CI / CD cycle will have to be started anew.

As already mentioned, a bug in production will cost you much more than a bug found during development.

The following tips will help improve process quality.

  • Slow down to speed up. Slow means uninterrupted, uninterrupted means fast.
  • Conduct a design review. The combination of code verification and requirements allows you to catch up to 70% of bugs.
  • Post code review. Well-tested code is 90% easier to maintain. One hour of reviewing saves you 33 hours of support. Programmers who conduct code reviews are 20% more productive.
  • Use the TDD approach. It reduces the number of bugs by 30-40 percent.
  • CI/CD. , , . , . CI/CD .
  • Increase test coverage . Your CI / CD process should run tests and stop if at least one of them crashes. This will help to avoid the deployment of bugs in production and save you a lot of money and time. Your goal is at least 70% coverage, but try to stay close to 80%. As you approach 100%, you will notice that coverage of important user processes with functional tests will give you more than a further increase in unit tests.

Conclusion


There are many ways to influence team performance, including:

  • Set realistic expectations
  • Track and control the number of open tasks
  • Control task sizes
  • Do not allow tasks to be accumulated by code review
  • Train developers
  • Provide a good balance between work and leisure
  • Implement effective development processes
  • Pay attention to employee retention

More articles:


All Articles