Examples of technical debt in implementing BI systems

The development and deployment of BI systems is a fairly quick and cheap process, but their maintenance over time is expensive. This can be imagined through the metaphor of technical debt.

Technical debt - denotes the problems accumulated in the program code or architecture related to neglect of quality in software development and causing additional labor costs in the future.

There are often well-grounded strategic reasons for taking on technical debt. Not all debt is bad, but all debt needs servicing. Technical debt can be paid by refactoring the code, improving testing, removing dead code, reducing dependencies, tightening the API, and improving documentation. The goal is not to add new functionality, but to make future improvements possible, reduce errors, and improve maintainability. Deferring such payments leads to complex costs. Hidden debt is dangerous as it increases silently.

Examples of technical debt in BI:

• There is a data warehouse in the BI project, but in fact it is a copy of the working database. As a result, storage benefits, such as data refresh rate, are lost, and data may be lost or damaged.
• When loading and updating data (ETL), the data is not checked / corrected. Errors are transferred to the application.
• Non-optimal field and variable names make editing and using the application difficult in the future.
• An initially improperly selected model / data structure of the application leads to problems during operation and modification of the application.
• The BI manager, under pressure, commits himself to complete the project by a certain date, which is more determined by the expectations of interested parties than by team resources or assessments (if any). And either this does not happen, or the project is of poor quality.
• Each BI project is its own island of user requirements and approaches. There is no centralized BI strategy. For example, due to the lack of unification, it takes more time to change different applications.
• The specific roles of people working on the project and the methods of their interaction (communication) are not defined.
• Administrators do not update BI software versions in a timely manner, as a result of which errors may occur, or some errors that have already been fixed by the vendor may be reproduced.

Only knowledge of this, unfortunately, does not provide any metrics for management. How to measure technical debt in a system or estimate the total value of this debt? The fact that the team is still working is not evidence of a low level of debt, since the full value of the debt becomes apparent only over time.

Some useful questions to consider:

1. How easy is it to fully test a completely new algorithm for calculating metrics?
2. How accurately can the effect of a new change in a system be measured?
3. How quickly can new team members get up to date?

We hope that this article can serve as an incentive for additional developments in the field of BI, including the improvement of testing methods, design patterns, and more. But the most important point is that technical debt is a problem that both programmers and managers should be aware of.