Kanban method: Example PNZ No. 1, the measurement-study process in a startup

In my  previous article, I described that we can perceive the delivery process as a Knowledge Accumulation Process .

The resulting process diagram shows how knowledge gradually enters and accumulates through a sequence of dominant activities. The points separating these activities are not a switch between functional specialists, but rather lead to a change in the pattern of interaction. An example of such a diagram was given for the average software delivery process.

Now it is time to show the application of this approach to other processes, and I will give some examples from a world that is not related to the supply of software. We will turn to the beginning of 2003 and my first cycles of creation-measurement-study, which we are now used to call a lean startup. At that time, I lived in New York and was one of the few engineers at the young online advertising company.

What do we deliver?

The end point of this process is confirmation from the business side of the relevance of the updated version of our product. Please note that we are not discussing the process of creating and delivering the product itself. The product has already been made and is ready to meet with its users. The question is: how do we build the learning process from now on?

Schematically, it may look like this:

In the early stages of this process, one activity dominates: ensuring that our vice president of advertising the next morning doesn’t have to shout: “Guy, where's my income ?!” This potential exclamation can be rephrased as a statistical hypothesis and tested on a small percentage of users, with a control group. We tried to prove that our functionality released in the latest version did no harm to the business. We wanted to be 100% sure that advertising campaign managers would be able to use the updated product, launch their portfolio of ads and get the same income per day as the control group using the previous version of the product.

Please note that this process has nothing to do with the regression testing that has already been carried out as part of the creation of the product. We could extrapolate our confidence in regression testing to our hypothesis, but don't assume one of the rules on Madison Avenue.

These experiments involved a chief product engineer (usually me), a vice president of advertising, a senior advertising campaign manager and an operations officer. When statistically significant results came, our confidence increased, activity began to fade, and new activity began to dominate.

Our next task was to measure the improvements from the new functionality, one of which was an increase in revenue from one user. New hypothesis: a test group using a new type of ad or a new algorithm will work better than the control group. A new experiment requires a change in the composition of the research team. We need a new type of advertising, so we brought in a graphic designer, user interface programmer and our creative director. An experiment can prove, for example, that a new geographic targeting algorithm will provide a higher percentage of clicks in the travel category. As the results arrive, we get confirmation that the product can be deployed to the entire user base, this activity also eventually comes to naught.

Then we faced the last check: will an improved product help with the proven effectiveness of its improvements in attracting more solvent customers? For example, at that time there was a regional airline that operated flights from one airport to several destinations. They would never buy a nationwide advertising campaign spending money, like American, United or Delta. Can our new product targeted at advertising with high targeting accuracy turn this airline into our customer? (So ​​it was.) This activity required the inclusion of a sales employee in order to work with the same advertising campaign managers and creative staff, while the role of engineers became minimal. Thus, we saw another change in the interaction pattern.

Again

In this example, we examined the delivery process in the field of professional services as a process of accumulating knowledge. We visualized it as a sequence of dominant activities. All once dominant activities disappear and give way to new ones. Each time this happens, a signal is formed not about transferring work to another functional department, but about a change in the interaction pattern.