Get best of the week

Like we did not blockchain

How did we use smart-contract to build a system for selecting the best technological projects in IT MTS? And what “traps” we fell into, but were able to get out, proving in the end that it is possible to maintain a distributed registry on mobile devices!



Why was a blockchain based system needed?


Let's start from the very beginning. MTS has a long tradition - to choose the best technological project made in a year and reward its team. The team receives prizes, respect and fame. Over the years, various projects have become winners: from highly loaded telecom systems to artificial intelligence systems.

The selection of the best project has always occurred in several stages:

  • Teams apply
  • Voting of respected technical experts takes place
  • After experts, projects are selected by managers
  • Upon completion of all stages, the big boss chooses the best project.

We decided that this scheme is not transparent enough for the participants and thought: why not give absolutely all experts in the company the opportunity to choose the best technological project? 


If we implement this opportunity right on the phone, we will see the current rating of projects and who votes for whom - this will ensure complete transparency of the process.

We read several articles about blockchain, and the idea of ​​building a distributed registry system firmly settled in our heads. But what if we apply smart-contract here ?

We were attracted by the following properties:

  • openness - there is no single server where you can manipulate information;
  • information placed in a distributed registry remains there forever;
  • information cannot be faked (well ... practically)

Not blockchain




Blockchain itself should not be used for such elections. But what if we take the protocols for building consensus in distributed systems and apply them to building consensus in human relations?

Given that we will have to work in an open network, we need to protect ourselves from non-Byzantine attacks and from the substitution of information on users' devices.

What are the alternatives


The most famous protocol is PAXOS. It lacks an explicit leader, and all changes go through a two-phase commit. At the beginning of each change, Propose occurs. If it was successful, then Accept is sent.


A feature of the algorithm can be called the fact that it uses global timers to determine which request was generated earlier and that the node making changes should communicate with all nodes of the network. Read more about the algorithm here .

The algorithm is used in many places, for example, in the Cassandra DBMS. We did not like this protocol in terms of the complexity of its implementation for the task. But the second option came up to us - it's RAFT. In fact, this is an evolution of the PAXOS protocol with a clear leader.

Simplified protocol can be described as follows:
· A network of devices is being built that know about each other;
· Devices choose among themselves a leader who makes “important decisions” (for example, adding an entry to the registry or changing the composition of the network);
· The leader device is responsible for the distribution of information throughout the network so that everywhere it is identical;
· As soon as the leader ceases to cope with his duties - choose a new leader.

Read about the protocol here .

Our implementation


What are we doing and why is the world another bike? 


Almost all users of our registry have mobile devices that are almost always on and almost always online, and actually use them almost always, so we decided to run the distribution registry algorithm on mobile devices, and not on the server infrastructure, like other well-known implementations .

Let's see what “traps” we got into, but we managed to get out ...

Trap number 1: “My address is not home and not street”


“Suddenly” it turned out that to build a P2P network that implements a distributed registry using the RAFT protocol for data replication, each device can communicate with each other, which means it is both a client and a server. Therefore, we need a public “white” IP address for each mobile phone (it may not be).

The number of real IPv4s is very limited, so telecom operators use NAT (Network Address Translation) technology in PAT (Port Address Translation) mode, translating several IP addresses of the internal network (which are distributed to subscribers) into one external public IP address. Thus, the ability to accept incoming connections from the Internet is excluded.

The good news is there is a lot of IPv6! 


We have IPv6 support included in the base package. Also, all modern phones support IPv6, and the operator assigns the subscriber a public “white” v6 address. Our choice is IPv6.

Trap number 2: Everyone went to sleep


Unlike servers, mobile phones still sometimes turn off. We found this when testing the first prototype. In addition, the IPv6 address provided by the operator is public, but not static, each new communication session is a new address. The address of the mobile device may change at any time. And if there is not a single phone with the address known to us on the network, it simply ceases to exist (there is nothing to connect to in order to “increase” it). Therefore, we had to violate our “no server” rule to some extent. We made one special node in the cloud with a static known address. Its task is to remember / update the composition of the network and not turn off. That is, it is an ordinary site, no one just votes from it, and turning to it, you can always get the current list of addresses of all network participants.

3: 



It was necessary to somehow solve the problem, so that not everyone could vote for projects, but those who needed it. The first idea was this: maintain a database of expert phone numbers. But they refused it, because they did not want to give the application the right to access this information.
As a result, they made everything simple: they decided to supply every voice, every registry entry with a digital signature on fashionable elliptical curves, which will determine the authenticity of the record. A WEB service was placed on the corporate network, which, by domain authorization, determined the expert and generated a unique QR code for him with the public and private encryption keys (of course, on the client side). The expert scanned the code from the application and connected. After that, the current current version of the registry “rolled up” on his phone and the opportunity to vote appeared.

4: Android 
 



During the testing, it was almost “suddenly” revealed that some users are the owners of well-known models of mobile devices running the iOS operating system. And it became clear that our registry software should run on different platforms. We looked towards the Kotlin programming language, which is not only “trendy, stylish, youth”, but also multi-platform .

The concept of multi-platform in Kotlin implies that there is part of the common and platform-specific code, but since the resources of our team are limited, we set ourselves the odious task of using a single version of the source code for all platforms! Of course, the executable module must be native for each platform. Kotlin was capable of this.

No sooner said than done! We have one single SourceSet with source code, from which we collect binaries for all platforms (!), Using the “depends” fitOn . Cool? Very cool!

Trap Number 5: Mobile traffic is not free 



How can we most effectively implement the interaction between network nodes so as not to spend all the subscriber’s traffic and not drain the battery of the mobile device? We assume that the network may consist of 1000 or more devices! The most obvious option is to use UDP instead of TCP, for example, in the “leader” selection procedure or when sending Heartbeats without data. UDP is more economical because it uses a simple data transfer model, without “handshakes” and confirmations. Fine! What else? Of course, asynchronous I / O!

We carefully read the documentation for Kotlin Native.

For all Unix or Windows based targets (including Android and iPhone) we provide the posix platform lib. It contains bindings to platform's implementation of POSIX standard.

Then we also carefully read the POSIX standard documentation and find an amazing function that allows us to process socket events in non-blocking mode! Having plunged headlong into the wonderful world of coroutine, sockets and C Interop, we were able to realize a very efficient transport. Super!

And in what form to send data? 



Of course CBOR!

A compact binary data format, which, by good luck, is implemented in the multi-platform library kotlinx.serialization . Just awesome!

Trap Number 6: Serialization 



This time, it really unexpectedly turned out that kotlinx.serialization is not under androidNative (under androidJvm, of course, there is). Dear colleagues from JetBrains have confirmed that at the moment they are not building a library for androidNative, and before the release of Kotlin 1.4 there is no longer any place for this task in the roadmap. 


What to do? If the mountain does not go to Mohammed, Mohammed goes to the mountain!

We ourselves have compiled kotlinx.serialization for all platforms, including androidNative! The most amazing thing is that it worked! 


Trap number 7: Where to store the log? 




Obviously, in the embedded key-value storage, but in which? We chose lmdbx for the compactness of the code, speed, multi-platform, and the lack of a WAL file. This library is developed by the guys from Positive Techlologies and originates from the legendary LMDB library from one of the authors of OpenLDAP Howard Chu. And that, in turn, is rooted in the implementation of B + tree from Martin Hedenfalk. By the way, out of the box, the library was not built for androidNative. We carefully collected all the errors, and the authors promptly provided fixes - for which special thanks to them!

Trap Number 8: C Interop 



Putting it all together proved to be a very non-trivial task. In addition to lmdbx and posix sockets, we integrated libraries for generating / validating digital signatures on elliptic curves and calculating SHA256 using the amazing C Interop mechanism . In simple words - from a native application on Kotlin, you can call the C-library function, including with pointers to pointers and other magic, it all looks a little strange.

For example, calling getaddrinfo to get sockaddr.



How do you like Ilon Mask?

Linking the C-library to the Kotlin Native executable is a separate quest, which we also managed to get through, but not without crutches. We dynamically form the def file directly in the build script to indicate the correct path to the libraries relative to the project root directory and then substitute its (def file) descriptor in the cinterops section. In the def file itself, only the absolute path is determined, which not only can have a different format if the assembly is performed under different OS, but actually on the local machines of the developers, of course, may also differ, which obviously leads to an error during the link.

About the election


The main elections we held during the day. A little over 20 experts took part in testing using our network. 21 projects were evaluated in 5 categories, that is, more than 100 entries with votes for projects were added to the register.

Conclusion


As a result of this small research project, we were able to prove that maintaining a distributed registry on mobile devices is possible! This opens up many possibilities for using this technology on IoT devices for organizing Edge-computing computing. Ahead of us are still waiting for load tests, vulnerability to attacks and fault tolerance. But we believe that we will succeed!

Authors of the article: architects and developers of the R&D Center of MTS Dmitry Dzyuba, Alexey Vasilenko and Semen Nevrev

More articles:


All Articles