Microservices - combinatorial explosion of versions

Hello, Habr! I present to you the author's translation of the article Microservices - Combinatorial Explosion of Versions .

In a time when the IT world is gradually moving to microservices and tools like Kubernetes, only one problem is becoming more noticeable. This problem is a combinatorial explosion of microservice versions. Nevertheless, the IT community believes that the current situation is much better than the Dependency hell of the previous generation of technologies. However, version control of microservices is a very complex problem. One of the proofs of this can be found in articles like “Return My Monolith to Me” .

If you are reading this text yet do not understand the problem, let me explain. Suppose your product consists of 10 microservices. Now suppose that for each of these microservices 1 new version is released. Only version 1 - I hope we can all agree that this is a very trivial and insignificant fact. Now, however, take a look at our product again. With just one new version of each component, we now have 2 ^ 10 - or 1024 permutations of how our product can be composed.

If there is still some misunderstanding, let me expand the math. So, we have 10 microservices, each receives one update. That is, we get 2 possible versions for each microservice (either old or new). Now, for each of the product components, we can use either of these two versions. Mathematically, this is the same as if we had a binary number of 10 characters. For example, let's say that 1 is the new version, and 0 is the old version - then one possible permutation can be designated as 1001000000 - where the 1st and 4th components are updated, but all the rest are not. From mathematics, we know that a binary number of 10 characters can have 2 ^ 10 or 1024 values. That is, we have confirmed the scale of the number with which it is dealing.

We continue the discussion further - and what happens if we have 100 microservices and each has 10 possible versions? The whole situation is becoming very unpleasant - now we have 10 ^ 100 permutations - and this is a huge number. Nevertheless, I prefer to describe this situation just like that, because now we are not hiding behind words like “kubernetes”, but we are encountering the problem as it is.

Why is this problem so fascinating to me? Partly because working earlier in the world of NLP and AI, we discussed a lot of the combinatorial explosion problem about 5-6 years ago. Only instead of versions we had separate words, and instead of products we had sentences and paragraphs. Although the problems of NLP and AI remain largely unresolved, we must admit that significant progress has been made over the past few years.(in my view, progress could be used to lshim if people in the industry a little less attention paid to the machine learning and other techniques a little more - but this is off-topic).

I will return to the world of DevOps and microservices. We are faced with a huge problem, disguised as an elephant in the Kunstkamera - because what I often hear is “just take kubernetes and helm, and everything will be fine!” But no, everything will not be fine if everything is left as is. Moreover, an analytical solution to this problem does not seem acceptable in view of the complexity. As in NLP, we should first approach this problem by narrowing the scope of the search - in this case, by eliminating obsolete permutations.

One of the things that can help - I wrote last yearabout the need to maintain a minimum spread between versions laid out for customers . It is also important to note that a well-developed CI / CD process is very helpful in reducing variations. However, the current state of affairs with CI / CD is not good enough to solve the permutation problem without additional tools for recording and tracking components.

What we need is a system of experiments at the integration stage, where we could determine the risk factor for each component, and also have an automated process for updating various components and testing without operator intervention - to see what works and what doesn’t.

Such a system of experiments could look like this:

1. ( — — ).
2. () CI — , CI
3. « » CI - , . , . , — .
4. CD , « » . .

Summing up, for me one of the biggest problems now is the lack of such a “Smart Integration System” that would link the various components into the product and thus allow you to track how the product as a whole is composed. I will be interested in the community’s thoughts on this (spoiler - I am currently working on the Reliza project , which could become such a smart integration system).

One last thing I want to mention is that for me a monolith is not acceptable for any project of at least medium size. For me, there is great skepticism about trying to speed up the implementation time and the quality of development by returning to the monolith. Firstly, the monolith has a similar problem of component management - among the various libraries of which it consists, but all this is not so noticeable and manifests itself primarily in the time that developers spend. The consequence of the monolith problem is the fact that it is impossible to make changes to the code - and the development speed is extremely slow.

Microservices improve the situation, but then the microservice architecture faces the problem of a combinatorial explosion at the integration stage. Yes, in general, we have moved the same problem - from the development stage to the integration stage. However, in my opinion, the microservices approach still leads to better results, and teams achieve results faster (probably mainly due to the reduction in the size of the development unit - or batch size ). However, the transition from monolith to microservices has not yet provided a sufficient improvement of the process - the combinatorial explosion of versions of microservices is a huge problem, and we have great potential to improve the situation as it is resolved.