👐🏾 🌛 👧🏽 How to get neighbors to work on their project, or InnerSource for a bank 🏥 🆎 🔌

What is development in Sber? In the eyes of an ordinary IT specialist: “This is where the code was written, go there!” But this has long been a stereotype, and they are not good. The rapid development of open source proves that such a culture has long become obsolete, and enterprise (if it is smart) has long revised the silo-based approach to development. Publishing all banking software in open source is an ~~effective way to commit suicide, a~~ rather controversial decision, and some kind of intermediate stage is needed. With the scale of the bank, we can launch our internal open source, rather than trying to check what we can show everyone and shake with fear for our little big secrets.

What is an InnerSource?

In 2000, Tim O'Reilly described the term inner source as a possible step towards a closed company entering the beautiful open source world. This is the application of best practices from open source within the corporation: from general openness and proactive mutual assistance to the formation of a market for the best solutions. The company continues to write proprietary code, but each of its employees has the opportunity to refine any internal system.

The advantages of InnerSource can be listed for a long time: reducing time to market, improving code quality, replacing external hiring with internal, improving cross-team interaction, and so on.

The difficulty lies in the fact that everything described above requires a change in the culture of communication between employees and teams, and this is not so easy to do in an organization that has worked differently for decades.

Of course, Sberbank is not the only company that has occurred to do something like this. In 2015, the InnerSource Commons community was created , which popularized the approach and removed the gap from the term to make it easier to find in search engines. This community has gathered the experience of dozens of companies and made effective recommendations for implementing InnerSource in your company, and holds an experience exchange conference twice a year. There are still many well-known technology companies that have been doing this ~~since the Pechenegs and Polovtsy~~ before it became mainstream.

In Russia, as far as we know, only one large retailer in the construction services market with a green logo is actively engaged in the intentional implementation of InnerSource, other companies either do not advertise their experience or do not participate in the community.

Each of these companies has its own positive and negative experiences. The overall conclusion in this case is very similar: the most delicious benefits can only be obtained with the participation of the vast majority.

Why would I collect it?

Almost every conversation about development in Sberbank comes to the question “how many programmers do you have there?”. Throughout the country, we have more than ten thousand of them, they are actively working on thousands of components, systems, libraries and others like them. As a consequence of such volumes, every day we have to solve the issues of managing dependencies on related teams, the relationship of leadership and the phases of Mercury.

At the end of the survey about what hurt the engineers, at the end of 2018 in Sber, it was decided to create a tribal SberWorks, which took over everything related to the processes and tools for producing software in the bank. The tasks and goals of the unit almost completely followed from the list of collected pains of developers.

I am ashamed to admit that the biggest pain at the end of last year was the lack of knowledge of what the neighbors in the workshop or even in the neighboring block in open space do. Because of this, we invented not only the wheel, but also two (substitute the desired number) of the same wheels in different units, both in different cities and in neighboring workplaces. As a result, having huge resources in the company, often instead of flying to Mars, our teams ran rakes, not knowing that the neighboring rakes had long been collected.

What to do with all this?

Time. To solve the issues of integration and finding the right people, create a single API registry for all systems of the bank with a large number of automation: call generation, stubs for the next version of the API, versioning and the like. Now this registry has turned into a separate cool product, which is definitely worthy of its article, but that's another story.

Two. Create a kind of "umbrella" with a single search engine over all engineering tools (JIRA, Confluence, Bitbucket, Nexus), combining the internal and external segments (yes, infamous alpha and sigma).

Three. Code, backlogs and artifacts, except those that explicitly forbid security to open, should be open to all company employees.

What was suggested?

In the process of communicating with developers, we identified the main reason for such a closed team within ourselves: the current ways of interaction between products. Any discussion of the refinement of related systems went according to one of three scenarios.

“Wait”

The most common way to interact. The adjacent team sets a deadline, it generally suits us, we put off the task deeper into the backlog.
In general, an acceptable option. But if the chain has dependencies on several teams, and the feature, as usual, needs to be displayed in production yesterday, then you need to look for compromises.

Compromises

Popularly, this method is called escalation. Take a more substantial manager with you and come to an adjacent team in order to move your task higher in the backlog. The disadvantages of this approach are obvious: most teams can play this card only a few times, after which the relationship between the teams deteriorates.

“Temporary permanent stub”

Saw your Frankenstein bike with crutches and temporary bazookas that duplicate the functionality of the system on which the addiction appeared. The saddest one, since it almost always remains for a long time (if not forever), generates duplication of code, and the team is forced to support a crutch solution.

We proposed a fourth way. Refine in the code base of the adjacent team under their sensitive supervision, reducing the execution time.

InnerSource

Upon completion of this interaction, team “A” from the picture receives precious feedback, nurtures expertise in an adjacent field and reduces the TTM of its refinement. Looking ahead, in the bank such interactions quickly acquired barter formats of various kinds: team “A” closes tasks from the technical debt of team “B” while they perform the necessary refinement.

Our way

At the very beginning, it seemed to us that we need to find places in which InnerSource will be in maximum demand right now. While we were thinking how to do this, without falling into a cycle of endless polls, the wise leadership appreciated the idea and proposed to ensure one hundred percent readiness of one hundred percent of Sberbank systems by the end of the year. We tensed up, our manager asked “what is one hundred percent?”, And the number of systems was reduced by about 20 times to modern and / or critical for business.

After that, the routine process of circulating this practice on teams began, at the end of which we saw a covered meadow with a list of repositories and rules for working with each of them.

We had a bunch of meetings at various levels, first with the technical leaders of the departments, then with team leaders and service owners, and finally with team members. We requested service representatives to provide information about the system itself, the development stack, and links to repositories, backlog, and documentation. At the same meetings, we were able to get first-hand pain information: endless backlog, lack of resources, old technology stack (for example, Delphi).

In the process of circulation, we have formed an understanding of the strong and weak links in the bank. There were very mature teams (for example, mobile development) that already work on the InnerSource principles on an industrial scale and shared a huge number of cones and approaches. But there were several teams (hello to the Pechenegs) that discouraged us by the lack of automatic tests, logging and code review practices.

Among our teams there is a huge cultural gap between “my military unit is working on the most correct tasks” and “let's do it together cool.” Most of the reactions were pretty monotonous: “we don’t want to take someone else’s code and then answer for it”, “we don’t want to give our developers because they will leave, but there are no resources anyway”. At this point, it was decided to invest in culture more than in technology:

HR : Google, , 10 % .
PR : Microsoft, open source.
: , - Delphi GraphQL, 10 % , !
, API, JIRA- .

With some error, we learned to track the inter-team interactions taking place in BitBucket and received information that we have about 30-50 new authors of InnerSource changes added every month. The figure in terms of the number of developers in the bank is not very impressive, but these are just improvements on business tasks.

Expectedly, data-driven changes in various integration buses and ETL services became very popular. This is easily explained by the large number of tasks and the low cost of improvements.

We carefully studied such improvements on the example of our ESB. A transition to a new solution is planned for her in the near future, so no additional resources were allocated to the team, while requests for revision only grew. At InnerSource, the guys saw salvation, quickly fidgeted and assembled a prioritizer who raises your task as much as possible if you are ready to do it yourself. In numbers, it looks like this: in October-November, almost half (101 of 209) requests for revision were completed by the teams themselves. This resulted in a four-fold reduction in waiting times from 2.5 months to 2.5 weeks.

Quite unexpectedly, designers took an active part, who are happy to help related teams when the latter lack resources or require a fresh look. As it turned out, there are quite a few teams that can utilize designers 100%, and they themselves are creative and love to change focus to some new product.

Afterword

The first steps in introducing transparency and interactions between teams in a company that is used to living the laws of enterprise are always the most difficult. We can safely say that the first walls have been broken through and a sufficient amount of successful InnerSource stories has been accumulated so that the movement inside the Sberbank will gain momentum.

The main challenge in the future we see as avoiding the Goodhart law . In any medium-sized and large-sized company, efficiency must be measured: come up with a number and make it grow. At a conference in Baltimore last fall, several cases were presented where setting goals for the readiness of teams and the number of improvements led to sabotage of metrics and burnout of movement leaders.

We believe that the resulting benefits completely discourage the invested effort, and openness in itself has countless advantages. We are ready to tell our path in more detail and exchange experiences, write-call - innersource@sberbank.ru . Kisses, Sberbank.