A shot from the film “Our Secret Universe: The Hidden Life of the Cell”

Investment business is one of the most difficult areas in the banking world, because there are not only loans, loans and deposits, but also securities, currency, goods, derivatives and all sorts of difficulties in the form of structural products.

Recently, we have seen an increase in the financial literacy of the population. More and more people are involved in trading in the securities markets. Individual investment accounts appeared not so long ago. They allow you to trade in the securities markets and at the same time either receive tax deductions or not pay taxes. And all the customers who come to us want to manage their portfolio and see real-time reporting. Moreover, most often this portfolio is multi-product, that is, people are customers of various business lines.

In addition, the requirements of regulators, both Russian and foreign, are growing.

To meet current needs and lay the foundation for future upgrades, we have developed the core of the investment business based on Tarantool.

Some statistics. Alfa-Bank’s investment business provides brokerage services for individuals and legal entities providing the opportunity to trade in various securities markets, custody services for the storage of securities, trust management services for individuals with private and large capital, securities issuing services for other companies. Alfa-Bank’s investment business is more than 3 thousand quotes per second, which are downloaded from various trading platforms. During the working day, more than 300 thousand transactions are concluded on the markets on behalf of the bank or its customers. On external and internal platforms, up to 5 thousand orders are executed per second. At the same time, all customers, both internal and external, want to see their positions in real time.

Background

Somewhere from the beginning of the 2000s, our areas of investment business have developed independently: exchange trading, brokerage services, currency trading, over-the-counter trading in securities and various derivatives. As a result, we fell into the trap of functional wells. What it is? Each line of business has its own systems that duplicate each other's functions. Each system has its own data model, although they operate on the same concepts: transactions, instruments, counterparties, quotes and more. And since each system developed independently, a diverse zoo of technologies arose.

In addition, the code base of systems is already out of date, because some products originated in the mid-1990s. And in some areas, it slowed down the development process, there were problems with productivity.

New Solution Requirements

Business realized that technological development is vital for further development. We were assigned the tasks:

Collect all business data in a single fast storage and in a single data model.
We must not lose or modify this information.
It is necessary to version the data, because at any time the regulator may ask for statistics for previous years.
We should not just bring some new, fashionable DBMS, but create a platform for solving business problems.

In addition, our architects set their conditions:

The new solution should be enterprise-class, that is, it should already be tested in some large companies.
The mode of operation of the solution must be mission critical. This means that we must be present at the same time in several data centers and calmly experience the disconnection of one data center.
. , , - . , .
, .

We went the standard way: formulated requirements and contacted the procurement department. From there we got a list of companies that, in general, are ready for us to do this. They told everyone about the task, and six of them received an assessment of the solutions.

We at the bank do not believe anyone’s word, we love to test everything on our own. Therefore, a prerequisite of our tender competition was passing stress tests. We formulated test tasks for the load, and already three out of six companies agreed at their own expense to implement a prototype of the solution based on in-memory technologies to test it.

I won’t tell how we tested everything and how long it took, I’ll summarize only: the best performance in load tests was shown by the prototype of the Tarantool-based solution from the Mail.ru Group development team. We signed a contract and started development. Four people were from Mail.ru Group, and from Alfa-Bank there were three developers, three system analysts, a solution architect, a product owner, and a Scrum master.

Next, I’ll talk about how our system grew, how it evolved, what we did and why.

Development

First of all, we wondered how to get data from our current systems. We decided that HTTP is quite suitable for us, because all current systems communicate with each other, sending XML or JSON over HTTP.

We use the built-in Tarantool HTTP server, because we do not need to terminate SSL sessions, and its performance is enough for us.

As I already said, we have all systems living in different data models, and at the input we need to bring the object to the model that we will describe at home. A language was needed to transform data. We chose the imperative Lua. We run all the code for data conversion in the sandbox - this is a safe place, beyond which the running code does not go beyond. To do this, simply do a loadstring of the necessary code, creating an environment with functions that cannot block anything or drop something.

After the conversion, the data must be checked for compliance with the model that we are creating. We discussed for a long time what a model should be, what language to use to describe it. We stopped at Apache Avro, because the language is simple and it has support from Tarantool. New versions of the model and user code can be put into operation several times a day, even under load, even without, at any time of the day, and very quickly adapt to changes.

After verification, the data must be saved. We do this with vshard (we have geo-spaced replicas of shards).

Moreover, the specifics are such that for most systems that send us data, it does not matter if we received them or not. Therefore, from the very beginning we implemented the repair line. What it is? If for some reason the object did not pass the data transformation or verification, then we still confirm receipt, but at the same time we save the object in the repair queue. It is consistent, located in the main repository with business data. We immediately wrote an admin interface for it, various metrics and alerts. As a result, we do not lose data. Even if something has changed in the source, if the data model has changed, we will immediately find it and can adapt.

Now you need to learn how to retrieve stored data. We carefully analyzed our systems and saw that on the classic stack from Java and Oracle there is always some kind of ORM that converts data from a relational view to an object one. So why not immediately give objects to systems in the form of a graph? Therefore, we gladly took GraphQL, which met all our needs. It allows you to receive data in the form of graphs, to pull out only what you need right now. You can even version the API with enough flexibility.

Almost immediately, we realized that the extracted data was not enough for us. We made functions that can be attached to objects in the model - in fact, calculated fields. That is, we attach a certain function to the field, which, for example, considers the average price of a quote. And the external consumer who requests the data does not even know that this field is calculated.

Implemented an authentication system.

Then they noticed that several roles crystallized in our solution. A role is a kind of aggregator of functions. As a rule, roles have different profiles of equipment use:

T-Connect: handles incoming connections, limited by processor, consumes little memory, does not store state.
IB-Core: transforms the data that it receives via the Tarantool protocol, that is, it operates with tablets. Also does not store state and can be scaled.
Storage: only saves data, does not use any logic. The simplest interfaces are implemented in this role. Scalable thanks to vshard.

That is, with the help of roles, we untied from each other different parts of the cluster that can be scaled independently of each other.

So, we created an asynchronous record of a transactional data stream and a repair queue with an administrator interface. Recording is asynchronous from a business point of view: if we are guaranteed to record data to ourselves, no matter where, then we will confirm this. If not confirmed, then something went wrong, the data needs to be sent. This is asynchronous recording.

Testing

From the very beginning of the project it was decided that we would try to instill test driven development. We write unit tests in Lua using the tarantool / tap framework, integration tests in Python using the pytest framework. At the same time, both developers and analysts are involved in writing integration tests.

How do we apply test driven development?

If we want some new feature, we first try to write a test for it. Having discovered the bug, we must first write to the test, and only then fix it. At first, it’s hard to work like that, there is a misunderstanding on the part of employees, even sabotage: “Let's quickly fix it now, do something new, and then cover it with tests.” Only this “later” almost never occurs.

Therefore, you must first force yourself to write tests, ask others to do it. Believe me, test driven development is beneficial even in the short term. You will feel that it has become easier for you to live. According to our feelings, 99% of the code is covered by tests now. It seems like a lot, but we have no problems: tests are run on every commit.

However, most of all we love stress testing, we consider it the most important and regularly conduct it.

I’ll tell you a short story about how we conducted the first stage of load testing of one of the first versions. We put the system on the developer's laptop, turned on the load and received 4 thousand transactions per second. Good result for a laptop. We put on a virtual load stand of four servers, weaker than in production. Deployed to a minimum. We start it, and we get a worse result than on a laptop in one thread. Shock content.

We were very sad. We look at the server load, and they turn out to be idle.

We call the developers, and they explain to us, people who have come from the Java world, that Tarantool is single-threaded. It can be effectively used by only one processor core under load. Then we deployed the maximum possible number of Tarantool instances on each server, turned on the load and received already 14.5 thousand transactions per second.

I’ll explain it again. Due to the division into roles that use resources differently, our roles that were responsible for processing connections and data transformation only loaded the processor, and it was strictly proportional to the load.

Moreover, the memory was used only for processing incoming connections and temporary objects.

On the contrary, on storage servers, the processor load grew, but much more slowly than on servers that handle connections.

And memory consumption grew in direct proportion to the loaded amount of data.

Services

To develop our new product specifically as an application platform, we made a component for deploying services and libraries on it.

Services are not just small pieces of code that operate on some fields. They can be quite large and complex designs that are part of the cluster, check the reference data, twist the business logic and give answers. We also export the service scheme to GraphQL, and the consumer receives a universal data access point, with introspection throughout the model. It is very comfortable.

Since services contain much more functions, we decided that there should be libraries in which we will take out frequently used code. We added them to a safe environment, after checking that this does not break anything for us. And now we can set functions for additional environments in the form of libraries.

We wanted us to have a platform not only for storage, but also for computing. And since we already had a bunch of replicas and shards, we implemented a semblance of distributed computing and called it map reduce, because it turned out to be like the original map reduce.

Old systems

Not all of our old systems can call us via HTTP and use GraphQL, although they support this protocol. Therefore, we made a mechanism to replicate data to these systems.

If something changes for us, peculiar triggers work in the Storage role and the message with the changes falls into the processing queue. It is sent to an external system using a separate replicator role. This role does not store state.

New improvements

As you recall, from a business perspective, we made asynchronous recording. But then they realized that it would not be enough, because there is a class of systems that need to immediately receive an answer about the status of the operation. Therefore, we have expanded our GraphQL and added mutations. They organically fit into the existing paradigm of working with data. We have a single point of both reading and writing for another class of systems.

We also realized that services alone would not be enough for us, because there are quite heavy reports that need to be built once a day, a week, a month. This can take a long time, and reports can even block the Tarantool event loop. Therefore, we made separate roles: scheduler and runner. Runners do not store state. They launch difficult tasks, which we cannot count on the fly. And the scheduler role monitors the schedule for launching these tasks, which is described in the configuration. The tasks themselves are stored in the same place as the business data. When the right time comes, scheduler takes the task, gives it to some runner, he considers it and saves the result.

Not all tasks need to be run on schedule. Some reports need to be read on demand. As soon as this requirement arrives, a task is formed in the sandbox and sent to runner for execution. After a while, the user receives an answer asynchronously that everything was calculated, the report is ready.

Initially, we adhered to the paradigm of saving all data, versioning and not deleting them. But in life, from time to time, you still have to delete something, mainly some raw or intermediate information. Based on expirationd, we made a mechanism for cleaning the storage of obsolete data.

We also understand that sooner or later a situation will come when there will be not enough space for storing data in memory, but nevertheless the data must be stored. For these purposes, we will soon make disk storage.

Conclusion

We started with the task of loading data into a single model, spent three months on its development. We had six data provider systems. The entire transformation code into a single model is about 30 thousand lines in Lua. And most of the work is yet to come. Sometimes there is a lack of motivation for neighboring teams, a lot of complicating the work of circumstances. If you ever face a similar problem, then the time that you think is normal for its implementation, multiply by three, or even four.

Also remember that existing problems in business processes cannot be solved with the help of a new DBMS, even if it is very productive. What I mean? At the start of our project, we created an impression among customers that now we will bring a new fast database and live! The processes will go faster, everything will be fine. In fact, technology does not solve the problems that exist in business processes, because business processes are people. And you need to work with people, not technology.

Development through testing in the initial stages can be painful and time consuming. But the positive effect of it will be noticeable even in the short term, when you do not need to do anything to conduct regression testing.

It is extremely important to carry out load testing at all stages of development. The sooner you notice some kind of flaw in the architecture, the easier it will be to fix it, this will save you a lot of time in the future.

There is nothing wrong with Lua. Anyone can learn to write on it: a Java developer, a JavaScript developer, a Python developer, a front-end or back-end. We even have analysts writing on it.

When we talk about the fact that we don’t have SQL, this terrifies people. “How do you get data without SQL?” Is that possible? ” Of course. On an OLTP class system, SQL is not needed. There is an alternative in the form of a language that returns you immediately document-oriented view. For example, GraphQL. And there is an alternative in the form of distributed computing.

If you understand that you will need to scale, then design your solution on Tarantool, immediately so that it can work in parallel on dozens of Tarantool instances. If you do not, then it will be difficult and painful, since Tarantool can efficiently use only one processor core.

How did we do the core of Alfa-Bank's investment business based on Tarantool