The IT world is diverse. Who doesn’t create any technologies and solutions, what doesn’t develop! Companies create products each in their own way, but many processes are similar, and therefore can be a useful experience for borrowing. So we thought: why not tell you about how we create our flagship Solar Dozor product? Our team is very experienced and energetic. Every day we have to solve non-trivial tasks, look for a killer of features and link the wishes of customers with our own roadmap. Suddenly, our experience is useful to someone?In general, we decided - we are launching a series of articles on how, where and under what circumstances our DLP system is born. All frankly, honestly, with photos and maybe even video proofs. And today you will find out where the creation of our product begins. Meet the Dozor Research Lab discovery lab.As you know, the theater begins with a hanger, and the development of any product - with an idea. One person came to another and shared his bright thought that came to mind. All. You can think of the research process in the Research Lab as running.But seriously, the history of technology says in the words of Steve Jobs: "Innovation distinguishes a leader from a catch-up." And Dozor Research Lab is the place where our innovative technologies are tested and nurtured. The key activity of our group is research (we call it foppish discovery). Discovery-process is at the junction of business, development, testing, implementation and marketing and is a necessary part of the workflow of each of these units. Today we will get to know some of the features of the Research Lab closer.We wanted to make the story lively and simple, so we immediately put aside the idea to use the language of business processes and any development process frameworks (we will talk about this in a separate article). On the contrary, we in a free style tried to highlight the remarkable aspects of the work of the Research Lab. And we got such a mind-map.
And now in order.Cloud of ideas
What is a sin to hide - we all love to dream. Only for the specialists of our group is a necessity. The search for new solutions and sensible ideas is impossible without proper imagination. Once David Hilbert said of his former student: “He became a poet. He had too little imagination for mathematics. ” This, of course, is irony. The creators of some of the existing technologies drew their ideas from fiction classics - writers and directors. The moral here is that the flight of thought must accompany both the work of the artist and the work of the researcher.In our work, we regularly discuss with colleagues, and, most importantly, with users, their own ideas and ideas. We give them the opportunity to dream: “What if you do such a thing? Or something like that. ” Some thoughts sound only once, and some at the same time from different sides. A so-called cloud of ideas is formed, from which no-no and life-giving moisture spills on the field of activity of the researcher.One of the researcher’s first tasks is to link seemingly unrelated ideas and suggestions into a system. From this, he must form a model (concept) that is beautiful in every sense. Let me explain this by the example of the formation of our solution for analyzing the behavior of UBA employees (User Behavior Analytics).For a long time we have been discussing with our colleagues in a close and wide circle our ideas and answers to the questions: What is such behavior in general? Is it possible to measure employee behavior? How can the behavior of a person or group affect information security? We discussed, someone gave presentations and conducted seminars, someone translated foreign articles and invented a prototype of a solution on his knee. Even tricks from video games were taken into account.It was rather unusual that, as a result of a series of discussions and rethinking, the mathematical model of behavior turned out to be very similar to the model of electron in quantum mechanics. This model of the object of the physical world contained the most suitable description of the necessary calculations (some of which relate to machine learning algorithms of the Anomaly Detection class). So, we can say that we are all a little electron.Call
The second important element of the discovery process is the challenge. The challenge to realize the boldest idea, to create a prototype of a complex concept, to achieve first-class quality. It is the challenge that motivates us to the result. If there is no call, then we can fantasize for a long time, program some separate parts, but we will not come to anything tangible in the end. Moreover, the challenge may be different - strategic, team, personal.One of our strategic challenges common to all the technologies that we create is to make them the best on the market. The challenge to our team is to show the effectiveness of the results of its work and to be in the lead of innovation in the company. My personal challenge is to usefully realize what I learned and adopted from my teachers.One way or another, the challenge gives drive to our work, each member of the team should feel it, otherwise the result will not be achieved.Respect for competitors
Many of you will probably be surprised now ... But I will call another feature of the discovery process this way: a sense of respect for competitors. We always analyze all the interesting technologies that our competitors create.It must be admitted that competitors from colleagues like our Research Lab also do not sit idly by. Analyzing their work, we compare their approaches with ours, see their good finds and shortcomings, and try to take them into account. It also happens that competitors offer interesting opportunities worthy of attention and mental “like”. Although sometimes defusing the situation with humor or a strong word about competitors will not hurt. For example, in situations where they begin to copy our developments and even terminology, and do it inconsistently and distort the meaning.Listen and do not give up
In the process of discussing new concepts very often have to face the resistance of their own colleagues. You need to be prepared for a variety of criticisms. A discussion with all interested parties forces you to adjust your own vision in advance, to prepare for uncomfortable, complex, and sometimes even wonderful questions. We are trying to develop such a skill.The legendary Steve Jobs was known for his love of cutting down ideas that people came to him with. Jobs's usual phrase “this is shit” should have been understood as “explain to me why this is“ the best way ”.This or that concept can seriously shake under the pressure of objections, but for a professional this is not a reason to give up. On the contrary, there is reason to think hard (and not sleep a couple of nights), change the conditions of the problem, add or remove something, and step by step come to the very same “the best way”. It should be understood that for the research and prototyping phase, the “Jobs reaction” is absolutely normal. But if you give up, you won’t get the result for sure.Here is another example from the development of UBA. We introduced a new concept - the "ego-network" of the employee. In accordance with the algorithm developed by us, those with whom he communicates face-to-face and on a regular basis fall into the ego-network of a person. There is also the concept of “private ego-network”, when such communication is conducted with recipients no longer known in the company. It can be either personal and related, or communications dangerous from the point of view of economic security.Our terminology was initially criticized by colleagues and technical writers. Other approaches to terminology were also considered. I had to conduct more than one brainstorming, several interviews with customers and security experts, pilot a prototype - all in order to make sure that of all the options, this turned out to be the most suitable and quickly fell into the users ’language.Experiment - First Judge
Experiments as part of the discovery process are necessary - most importantly, do not be afraid to experiment. No one will come and tell you exactly how to check the functionality. You have to figure out for yourself how to test the model for strength - this is akin to car crash tests. 
For example, how did we experiment with image recognition technology in images? They took the usual seal of our company and slapped it on all kinds of documents, on images or even on hand. Then printed, scanned, photographed, studied all its possible variations and distortions. And they sent the resulting samples to our recognition tool. Of course, we then prepared a representative sample of initial examples. But usually it all starts with the widest and fastest experiments.An honestly set experiment is a powerful help in the initial assessment of risks and hypotheses.PS Growing Talents
This part of our work can be considered secondary to the main activity. But in fact, it is very important for the whole company. It happens that employees come to our laboratory who don’t imagine what exactly in IT they would like to do. For example, with education in the field of information security, but without any programming skills. And in 1-2 years, with the help of colleagues and own persistence, a person from scratch masters programming and becomes one of the best developers in his field.How is this done? Just as part of our discovery process, we immediately after employment offer beginners a pool of tasks with a different plan, if possible. This allows you to identify the strengths of a person and greatly increases his effectiveness. Our group has enough opportunities for self-development. We try to build work so that tasks do not put pressure on a person, but allow him to grow.Organization of the discovery process using Solar Dozor UBA as an example
We talked about the individual details of our Discovery puzzle - now let's try to look at it from the side and describe the main stages of the research activity. As an illustration, I will tell you how the process of researching our new Solar Dozor UBA product from the class of UEBA systems went through.A few words about the duration of the discovery process
Even before the development of the Solar Dozor UBA behavior analysis module, the discovery process took a little more than a year. Time, on the one hand, is considerable. On the other hand, a bold initial concept resulted in a considerable system of functions, each of which had its own business justification. Therefore, a rush could destroy this system.At the start of research within Russia, there were no confirmed examples of working domestic solutions of the UEBA class. Therefore, first of all, we relied on the experience and significant scientific publications of foreign developers of such solutions, which also only gain momentum and often offer very different tools and frameworks. There was no talk of any “traditional” set of data analysis technology, and we could not just come to the customer with beautiful ideas.Deep dive
Details. . .
, , . ( Research Lab) . – . , , , .
At the same time, we discussed our ideas within the team - held regular meetings with team leads, system architects, business analysts. There was a collection of opinions and criticism. One cannot fail to note the determining position of the leadership on key issues. So the model of the studied object (human behavior) was born and a circle of technologies capable of realizing it was outlined. The working group made a working prototype of the solution.Details of the first prototype
Case Study Solar Dozor UBAPython, Pandas, -, Plotly. PosgreSQL.
, -, . . , . Anomaly Detection, LOF-. . Solar Dozor, Solar Dozor UBA.
Naturally, at the stage of piloting the prototype, not everything goes smoothly. It is important to build communications and win high customer loyalty. It is important to find and assemble a team of responsible and qualified colleagues from the customer who are ready to work together. To do this, you first need to take care of the pilot's planning, its goals, objectives and benefits for each team member.Pilots and trial operation of the disposable Solar Dozor UBA prototype took a long time in the total discovery duration. At the same time, they gave us the opportunity to make important adjustments in the system of user-defined functions and in our mathematical model. We were able to leave part of the functions in which we became confident, and at the same time discarded some of our unconfirmed hypotheses. Also, the actual operation of the prototype revealed new opportunities and areas of interest for users. We would like to thank the customers who were passionate about our technology who agreed to test the solution and gave us such a useful response!Not all technologies are successfully tested, and this is normal.Examples. , . . DLP-, , Solar Dozor , , , . , , . , . .
- , . . «» , . .
, , . , , discovery. , , , .
Upon completion of the trial operation of the prototype, we conduct an interview with the first users, preferably with a visit to the customer. Given that we are talking about new features and functions for the user, it is extremely useful to make such contacts with the participation of business analysts and direct prototype developers. The fact is that at this stage the price of an error to misinterpret the words of the customer is very high and can lead away from the best solutions. And, of course, all communications must be logged. This helps in many controversial situations and may subsequently affect the determination of the minimum useful functionality (MVP) of the first version of the product on the market.Accordingly, at the finish line, the key result of the research work is a conceptual proposal describing the minimum useful functionality. A review of all reports and prioritization of certain features and functions is carried out. At this time, it is necessary to widely publicize these results for interested colleagues and management. Even very good results without the proper involvement of colleagues from development, business and marketing can go unnoticed or not fully understood.Here, perhaps, are all the main points that accompany the research process in our Research Lab, regardless of the size of the study, large or small. If the research is successful, and the game is worth the candle, then the development department picks it up. A separate development team can already stand out here. The main product development processes based on the Agile standard according to the Scrum method are launched. But this is the next story rich in interesting details.
Text writer:Maxim Buzinov, Head of Dozor Research Lab.Illustrations:Anna Yakovlenko, Data Analyst.