How to build a Tier IV data center according to the N + 1 scheme

UPS systems with isolated parallel bus (IP-Bus) - the answer of developers to the growth of data center capacities. Many data centers with IP-Bus have already been built in the world, including those with the Tier IV Uptime Institute certificate. Such decisions are being watched by Russian customers.

In the practice of building data centers there is a steady trend towards their enlargement. Objects with a capacity of 100 MW have appeared in the world. Russia also does not stand aside, although it follows in this direction with some delay. 10 years ago in our country, a data center with a capacity of 5 MW was considered large, and today several leading operators have announced plans to build facilities for 2,000 or more racks, which corresponds to an energy consumption of 15 MW and above.

For the organization of engineering systems of high power, as world practice has shown, the most expedient from an economic point of view is a circuit with parallel N + x (N + 1, N + 2 ...) connection of devices. Moreover, the unit capacity of the world's largest UPS installations - dynamic, which can be used in such solutions, is limited by the power (and cost) of the largest diesel engines used to work with UPSs.

However, the direct parallel connection of the UPS, providing the ability to create effective N + x configurations, has a number of significant drawbacks:

• Low-voltage installations can only be used in systems up to 5 MW. This is due both to restrictions on the available ratings of low-voltage package devices (6300 A), and to high short-circuit currents, the values ​​of which can exceed 150 kA.
• Medium voltage solutions, which make it possible to build systems of more than 5 MW, increase the cost of the power system and do not always suit customers in terms of operation.
• Common components of the system - input and output buses, bypass - are common points of failure.

The N + N (2N) scheme, corresponding to the level of fault tolerance of the Tier IV Uptime Institute, makes it possible, by building separate energy modules, to get away from the main disadvantages of classical parallel systems. But this approach has other obvious disadvantages:

• 100% duplication of equipment, i.e. high capital costs;
• large footprint;
• maximum load level - 50% (in practice - not higher than 40%);
• high operating costs.

For these reasons, the N + N (2N) configuration is rarely used for facilities with a capacity of more than 10 MW.

In 2005, a solution was found that, while maintaining the main advantage of the parallel circuit — the optimal number of UPS modules in the N + x circuit — to put into practice systems with a capacity of up to 20 MW while remaining at a low voltage of 0.4 kV. This solution, dubbed the IP-Bus configuration, meets the highest level of fault tolerance (Tier IV Uptime Institute). The IP-Bus idea is based on the use of a ring bus to connect individual UPS modules, each of which is isolated using a reactor (Fig. 1).


Fig. 1. UPS Isolation-Parallel

In IP-Bus systems, each UPS operates on its own load output and is simultaneously connected to a common bus (IP-Bus) through an isolation choke, which performs several important functions:

• allows you to redistribute active power between UPSs - a UPS module with a lower load “helps” other modules by transmitting excess power through an IP bus (Fig. 2);
• provides uninterrupted power supply to the load in case of shutdown of the UPS for maintenance work or in case of an accident (Fig. 3, Fig. 4);
• slows down the exchange of reactive currents between UPS installations, due to the impedances of reactors, so that there is no need to control reactive power inside the system.

• (. 5).
  
  - IP-Bus, , N + (N + 1, N + 2…). — 70%, .

<img src = " " alt = "image" />
Fig. 2. An example of load distribution in a system of 16 UPS installations

In contrast to the “direct” parallel configuration, in the IP-Bus system, each UPS installation controls its output voltage independently of the others - there is no centralized control device and a common point of failure is eliminated. Assuming that the power flow from one UPS suddenly disappears for some reason, its load remains connected to the IP-Bus using the IP choke, which now works as a backup power source. In this scenario, the load will automatically and uninterruptedly receive power from the IP-Bus (see Fig. 3).


Fig. 3. Example of system redundancy in case of failure / shutdown of one UPS installation

In practice, the IP-Bus is usually made in the form of a ring, as shown in Fig. 4. The second segment of the IP-Bus, often called the Return-Bus, acts as a backup source for loads, allowing you to directly connect them to the IP-Bus through separate switches - a kind of bypass, which ensures the rated voltage on the load even in emergency situations or when performing service work. Such bypasses are not a common point of failure, because at the initial moment of time, until the bypass closes, the load continues to receive power from the IP-Bus via the IP choke, as mentioned above.


Fig. 4 Example of operation of load No. 2 directly from the backup IP Bus

The behavior of the IP-Bus system under short-circuit scenarios also differs significantly from processes in a “direct” parallel configuration. In the IP-Bus circuit, possible short circuits due to the use of IP chokes have only a negligible effect on the loads. In this case, short-circuit currents do not exceed 100 kA, which allows the use of standard switching, protective and bus equipment.

In the event of a short circuit on the load side of the UPS (see Fig. 5), the effect of such a short circuit on the entire system is relatively insignificant due to the fact that the remaining loads are isolated from the UPS by means of two reactors connected in series. On the other hand, the short-circuit current supplied by the UPS to a common IP bus is limited by the resistance of the IP choke. Therefore, changes in voltage on unaffected loads are insignificant and remain in the safe region of the ITI curve (CBEMA).


Fig. 5. Example of distribution and values ​​of short-circuit currents in the IP-Bus system with short-circuit on the load power bus connected to UPS 2

In the event of a short circuit on the IP-Bus, only one IP reactor is located between the fault point and the UPS or the load. Therefore, the voltage drop across the loads in this scenario will be much larger compared with the short circuit in the load distribution system. With a low transition resistance, the initial voltage drop across the load will be 30%. For sensitive server power supplies, according to the ITI curve (CBEMA), this voltage drop is acceptable for a maximum of 500 ms. The use of segmented directional protection, specially adapted to the requirements of the IP-Bus system, allows you to clean the short circuit on the IP-Bus for 60 ms by selectively isolating the short circuit and at the same time allows the part of the IP-Bus system that is not directly affected to remain completely workable.

The IP Bus system consists of several UPS installations, the number of which is determined by the set N + x redundancy level and includes the following main components: a UPS with a power storage device, an IP choke for connecting the UPS installation to the IP bus, and switches necessary for safe operation of the system.

In fig. Figure 6 shows one embodiment of an IP Bus system based on a rotary UPS.

System elements:

1. External network
2. IP-Bus
3. IP-Return-Bus
4. Rotary UPS with a flywheel
5. DGU for a long network interruption (optional)
6. IP choke
7. Bypass switch
8. IP -switches
9. Load




Fig. 6. An example of an IP-Bus system using a rotary UPS Piller UNIBLOCK and an external DGU with a “lower” turn-on.

According to Piller's practical experience, dynamic UPSs with flywheels (Fig. 6) as backup energy storage devices are ideal for IP Bus systems since Kinetic drives as part of the DIBP can operate in the mode of both instantaneous energy absorption and instantaneous discharge, which is important for stabilizing the operating parameters of the IP-Bus system when the load changes.

In addition, the motor generators in the DIBP have the ability to supply high short-circuit currents of up to 20 x Inom, which allows IP-Bus systems to cope with short-circuit cleaning for a very long time without exposing neighboring loads to the negative effects of a short circuit.
Static UPSs with batteries have limited ability to instantly send and receive high currents, and in addition, the short-circuit currents of the UPSs themselves are relatively low. For these reasons, IP-Bus solutions on static UPSs are more of an experiment, and they are practically not found in existing data centers.

The world's first IP-Bus system was implemented in 2007 for a 36 MW data center in Ashburn (Virginia, USA). Two separate IP-Bus systems were installed at the facility, each of which includes 16 Piller UNIBLOCK UBT 1670 kVA UPSs with flywheels in a 14 + 2 configuration. In case of long-term outages of the external network, each DIBP is reserved by a separate 2810 kVA diesel generator with a “lower turn-on”, which works both on uninterruptible and guaranteed power supply loads
Following the success of the first IP-Bus system, this configuration quickly gained popularity in the data center industry. Another milestone in the development and recognition of IP-Bus technology was the receipt of the Tier IV Design & Facility Uptime Institute certificate in September 2017 by the Australian NEXTDC B2 data center with IP-Bus N + 1 power supply system.

The Russian data center market is only entering the phase of the construction of large facilities with a capacity of over 10 MW. Based on the results of the first concept calculations and budget assessments of IP-Bus solutions at several data center projects in Russia (in the range of capacities of 5-15 MW), the following conclusions can be drawn. Compared with the 2N configuration on static UPSs, IP-Bus solutions based on DIBP are not more expensive in initial capital costs, they give 30-60% gain in occupied space, more than 50% more profitable in terms of cost of ownership (TCO) for a period of 10 years. Compared to the N + 1 distributed redundant configuration (DR 3/2, 4/3), implemented on both static and dynamic UPSs, IP-Bus solutions based on DIBPs are not more expensive in initial capital costs (for data centers with a capacity of 10 MW or more), give a gain of 20-50% in the occupied area, 50% more profitable in terms of TCO for a period of 10 years.

Thus, I am sure that the implementation of IP-Bus systems in Russian data centers is only a matter of time.